ACE-A1.2 testing - Arista Certified Engineering Associate Updated: 2023

Testing multiple devices in parallel using the same ATE results in reduced test time and lower costs, but it requires engineering finesse to make it so.

Minimizing test measurement variation for each device under test (DUT) is a multi-physics problem, and it’s one that is becoming more essential to resolve at each new process node and in multi-chip packages. It requires synchronization of electrical, mechanical and thermal aspects of the whole test cell so that chipmakers can ensure variation is confined to the DUTs. This assumption is vital when applying statistically determined test limits that appropriately adapt to local process variation.

The test world is not perfect, which necessitates accounting for differences in test measurement environment. Getting this right can have a big impact on an IC product’s outgoing quality and reliability.

Fortunately, the test data from each test site can be used to determine these differences. Armed with this knowledge, engineers can adjust their statistically based pass/fail criteria algorithms accordingly. As a result, both product yield and quality improve. This is essential because parallel device testing continues to increase at wafer and unit level testing. And the end products are being used across a variety of markets, including data centers and safety-critical applications such as automotive, which continue to demand escape rates on the order of 10 ppm and lower.

Unit-level test includes final test, burn-in modules, and system-level test (SLT). But it’s wafer and final test that pose the more daunting technical challenges due to the smaller test interface boards for probe cards and loadboards, respectively.

“Our customers’ wafer probe cards are growing in their usage of multi-site test,” said Keith Schaub, vice president of technology and strategy at Advantest America. “Combine this with an increase in DUT pins for some products (large digital devices), and the common concerns of probe card planarity and the probe tip damage, due to burning when too much current is applied, become an even greater concern.”

Fig. 1: Progression from 1X to 16X site testing. Source: Anne Meixner/Semiconductor Engineering

“You could probably drive a truck through this range,” said Mark Kahwati, product marketing director for semiconductor testing group at Teradyne. “There are some applications where it remains single-site. Then consider controllers used in automotive safety, airbag controllers, and ABS controllers. You see anywhere from 4 sites to maybe 8 to 12 sites. Then, with relatively low pin count devices in automotive, the number of sites approaches 64 sites in parallel, if not more.”

While the same economics drive the increasing the number of devices tested in parallel, those numbers can vary greatly by industry sector and device type (see figure 2 below).

Fig. 2: Number of sites per test insertion, by industry and device type. Source: Teradyne

In parallel test, every effort is made to minimize ATE and associated test hardware differences between each test site. With their latest ATEs, vendors provide new capabilities to support multi-site testing with increased attention to reducing the test hardware contributions. Analog test measurements require more care in the design of path from the automatic test equipment (ATE) hardware to the DUT, but ATE instrumentation can be calibrated to account for differences along these paths.

Nevertheless, differences persist. And when applying statistically based outlier detection techniques, these differences matter. Engineering teams at Texas Instruments, AMS AG and Skyworks Solutions have documented the impact of site-to-site differences. In their 2015 DATA workshop paper, engineers from Skyworks Solutions and Galaxy Semiconductor articulated why it matters:

“It would therefore be logical to assume that adjacent columns or rows of devices should show nearly identical data distributions. However,” they wrote, “a tester with multiple test site hardware components will show systematic variation from one test site to the other…In spite of the best efforts to ensure that test hardware is consistent from one piece to another, measurable biases often emerge. These biases can and do contribute to variation in the statistics behind NNR values. These biases, because they are consistent and predictable, can be managed with a linear offset applied to the measurements.”

Test limits based upon the statistical techniques have become a common tool in a product engineer’s toolbox. Such techniques inherently assume that all die/units have the same measurement environment. As a result, when testing devices in parallel, engineering teams first focus on achieving that assumption.

Reducing test cell site-to-site variation
Any measurement system has sources of error. For semiconductor device testing, both the signal paths and the power paths between ATE and DUT need to be considered. At each hardware device and connection there exists a tolerance for each measurement parameter. For instance, edge placement accuracy represents a timing tolerance for pin electronics cards. These tolerances add up between the path of the DUT pin/pad and the ATE instrument.

Fig. 3: Contributions to measurement errors in the test path. Source: Anne Meixner/Semiconductor Engineering

For first-order understanding, the physical area of a device board/probe head combined with a device’s pin count factors into the amount of parallelism that is physically possible. Next, the mechanical, thermal, and electrical attributes of the test cell need to be understood, as all of them can contribute to errors.

Reducing these contributions to measurement error meets the overall goal of having a high accuracy test set-up. With multiple sites come a few unique challenges to meet equivalence between sites. Engineering teams need to:

• Balance the thermo-electrical challenges across the multiple sites;
• Manage the test cell resources to deliver identical voltage and currents;
• Design ATE instrumentation to stringent specifications to reduce tester channel differences in site to site;
• Include calibration techniques to reduce signal path variation, and
• Design test interface boards — also known as probe cards and loadboards — to assure equal transmission line lengths and environment, such as coupling to nearby signals.

“At wafer test there are a number of items impactful to site variation — mechanically, how the probes contact the pads, contamination on the pad or the probe, and temperature variation across the wafer/chuck,” said Darren James, technical account manager and product specialist at Onto Innovation. “Electrically, design and layout of the interface and the probe card to provide good impedance matching of the sites/pins is especially important if resources are shared between sites. Interface design also will impact the amount of cross-talk and leakage.”

From an package test perspective George Harris, vice president global test services at Amkor Technology, noted several commonly observed causes for site-to-site test variation:

• Routing on the board impacting resistance, capacitance, inductance, coupling and crosstalk variations between sites;
• Thermal differences across the test boards, both on top side and backside;
• Differences in tester resources between channels.

“It’s always best to design and characterize the production test environment relative to the product specification requirements,” Harris said. “Even fairly simple products pushing the test environment with many sites tested or stressed in parallel may have power distribution differences, as will a complex SoC.”

Identifying and dealing with site-to-site variation
Testing cuts across multiple processes as it shifts both left and right. As a result, variation needs to be dealt with in the context of other processes. For example, during test, engineering teams need to identify excursion-based site-to-site variation to which they can respond. In contrast, product engineering teams may need to account for site-to-site variation when applying their pass/fail criteria.

“Engineering teams need to have a test process control system in place, with analytics to assist with the root cause of variance when issues like site-to-site variation are detected,” said Greg Prewitt, director of Exensio solutions at PDF Solutions. “The control system needs to be able to alarm/alert quickly so the team can take action to resolve the situation before material needs to be scrapped. Some of the best practices include automated responses, such as clean probe needles, or activation of an Out of Control Action Plan (OCAP) processes, which in turn needs to be integrated with manufacturing execution systems (MES) for automated holds on suspect lots.”

When parallelism equals a whole wafer touch down, engineers need to consider more advanced statistics as they crop up. Consider, for example, smart card devices pushing 4K sites at wafer test.

“Big probe heads for this many sites raise challenges of temperature variation across the chuck, which could impact device temp sensor measurements if not managed,” said Ed Seng, product manager of digital segments at Teradyne. “Site-to-site correlation at these high counts has to be done more statistically, relying on a higher volume of data as compared to single stepping a single die across a wafer.”

With 4K site comparisons, the correlation analysis becomes way more complicated than 4 or 8 sites.

So exactly how is site-to-site variation analyzed? Engineers can use gauge R&R techniques to assess repeatability and reproducibility across multiple sites. For 2X to 16X parallelism, analyzing site-to-site variation easily can be supported by most statistical software packages (e.g. JMP, R).

Factories can respond to tester hardware differences that require preventive maintenance. Yet the subtle differences in that signal path, from instrument to DUT pad/pin, add up. The latest ATE models have been designed to minimize such differences. Also, test interface boards — such as probe cards and loadboards — must be designed with expert knowledge in PCB technology in order to minimize differences.

But in both wafer and unit-level test factories, the reality is there exists a lot of older ATEs. As a result, the latest products may be tested on older equipment. That, in turn, can result in site-to-site test result differences. If the differences are minor and the test process is well controlled (i.e., a Cpk greater than or equal to 1.33), the impact to device yield and quality will be negligible.

The definition of negligible though changes when applied to sensitive analog measurements coupled with the application of statistical outlier detection test criteria.

Outlier detection tests range from the simple part-average testing (PAT) to the sophisticated Near Neighborhood Residual (NNR) testing. When required, these data analytics-based test techniques can accommodate the observed site-to-site variation. In fact, it becomes a necessity, as illustrated by two examples for how engineers accommodate for it. The first example looks at RF test and PAT. The second looks at IDDQ wafer test and NNR.

“For an RF device, we ran into a similar problem where devices were tested in quad sites. We had one site that gave statistically different tests results from the others. With RF, it’s very difficult to match four sites very well. The RF performance characteristics on four sockets, four contactors, and four sets of components are going to be different,” said Jeff Roehr, IEEE senior member and 40-year veteran of test. “If we didn’t account for that, we would have a very wide distribution in the test data, which made it hard to see the outliers. We learned over time that we had to analyze test data on a per-site basis. In effect, we had four sets of software running simultaneously doing PAT.”

With device populations on the order of hundreds to thousands, engineers establish PAT and Dynamic PAT limits. On smaller statistical populations of about 25 to 40, like those used for Z-PAT and NNR, the impact of test hardware site-to-site becomes more noticeable. Especially with sensitive analog measurements, neglecting the impact can result in failing good die, as well as passing bad die.

Over the past decade, several papers describing outlier detection techniques have stated that test hardware site-to-site variation impacts the ability to precisely discriminate between good and bad die. A 2016 Texas Instruments paper noted that site-to-site variations need to be accounted for when applying NNR techniques. And a 2018 AMS AG paper on adaptive test for mixed-signal ICs included site-to-site variation in its dynamic PAT limits.

In a 2015 DATA workshop paper, engineers from Skyworks Solutions and Galaxy Semiconductor presented a method to offset site bias when applying NNR. For each test measurement, they shared a technique for calculating each site’s bias To illustrate their technique assume 4X testing and a test called ACB22. The calculation follows:

1. Calculate the medians for test ACB22 for sites 1, 2,3, 4: ACB22Med (site 1)
2. Calculate the mean of these four medians: Mean of ACB22Med
3. Site bias for site 1 equals Mean of ACB22Med minus ACB22Med (site 1)

Applying the resulting site bias to the NNR limits more precisely discriminates between good and bad die.

Conclusion
With the continuing cost pressures to test semiconductor devices in parallel comes the engineering effort to create the measurement environment so each site is equivalent.

“The economic motivation for higher multi-site test still holds,” said Teradyne’s Seng. “The same type of multisite challenges exist as they have in latest generations but continue to grow into the next degrees of technical complexity,” said Teradyne’s Seng. “Most of the challenges are in the device interface area, from the tester instrument device interface board (DIB) connection through to the device connections. The best test systems will take care of all the other multi-site factors and make it fast and easy to implement high multi-site test solutions.”

Still, not all engineers get to test their products on the best test systems. With products that are tested in parallel, they need to manage their current products with the test equipment they have in their factories. This requires them to reduce site-to-site variations in test processes as much as possible with design and necessitates them to respond to excursions associated with the realities of a factory floor. In addition, inherent site-to-site variation needs to be considered when product engineers use statistically based pass/fail test limits. Fortunately, the test data can be used to discern the differences between test hardware and DUT contributors.

Parallel test execution reduces overall test cost. Yet the simplicity in a diagram of testing four units belies the engineering effort behind it.

Related Stories
Geo-Spatial Outlier Detection
Using position to find defects on wafers.

Part Average Tests For Auto ICs Not Good Enough
Advanced node chips and packages require additional inspection, analysis and time, all of which adds cost.

Chasing Test Escapes In IC Manufacturing
Data analytics can greatly Boost reliability, but cost tradeoffs are complicated.

Adaptive Test Gains Ground
Demand for improved quality at a reasonable cost is driving big changes in test processes.

The Civil, Architectural, and Environmental Engineering Department laboratories provide students with fully equipped space for education and research opportunities. 

Structural and Geotechnical Research Laboratory Facilities and Equipment

The geotechnical and structural engineering research labs at Drexel University provide a forum to perform large-scale experimentation across a broad range of areas including infrastructure preservation and renewal, structural health monitoring, geosynthetics, nondestructive evaluation, earthquake engineering, and novel ground modification approaches among others.

The laboratory is equipped with different data acquisition systems (MTS, Campbell Scientific, and National Instruments) capable of recording strain, displacement, tilt, load and acceleration time histories.  An array of sensors including LVDTs, wire potentiometers, linear and rotational accelerometers, and load cells are also available.  Structural testing capabilities include two 220kips capacity loading frames (MTS 311 and Tinius Olsen), in addition to several medium capacity testing frames (Instron 1331 and 567 and MTS 370 testing frames), two 5-kips MTS actuators for dynamic testing and one degree of freedom 22kips ANCO shake table.  The laboratory also features a phenomenological physical model which resembles the dynamic features of common highway bridges and is used for field testing preparation and for testing different measurement devices.  

The Woodring Laboratory hosts a wide variety of geotechnical, geosynthetics, and materials engineering testing equipment.  The geotechnical engineering testing equipment includes Geotac unconfined compression and a triaxial compression testing device, ring shear apparatus, constant rate of strain consolidometer, an automated incremental consolidometer, an automated Geotac direct shear device and a large-scale consolidometer (12” by 12” trial size). Other equipment includes a Fisher pH and conductivity meter as well as a Brookfield rotating viscometer. Electronic and digital equipment include FLIR SC 325 infrared camera for thermal measurements, NI Function generators, acoustic emission sensors and ultrasonic transducers, signal conditioners, and impulse hammers for nondestructive testing.

The geosynthetics testing equipment in the Woodring lab includes pressure cells for incubation and a new differential scanning calorimetry device including the standard-OIT.  Materials testing equipment that is available through the materials and chemical engineering departments includes a scanning electron microscope, liquid chromatography, and Fourier transform infrared spectroscopy.

The Building Science and Engineering Group (BSEG) research space is also located in the Woodring Laboratory.  This is a collaborative research unit working at Drexel University with the objective of achieving more comprehensive and innovative approaches to sustainable building design and operation through the promotion of greater collaboration between diverse sets of research expertise.  Much of the BSEG work is simulation or model based.  Researchers in this lab also share some instrumentation with the DARRL lab (see below). 

Environmental Engineering Laboratory Facilities and Equipment

The environmental engineering laboratories at Drexel University allow faculty and student researchers access to state-of-the-art equipment needed to execute a variety of experiments. These facilities are located in the Alumni Engineering Laboratory Building and includes approximately 2000 SF shared laboratory space, and a 400 SF clean room for cell culture and PCR.

The major equipment used in this laboratory space consists of: Roche Applied Science LightCyclerÔ 480 Real-time PCR System, Leica fluorescence microscope with phase contrast and video camera, Spectrophotometer, Zeiss stereo microscope with heavy duty boom stand, fluorescence capability, and a SPOT cooled color camera, BIORAD iCycler thermocycler for PCR, gel readers, transilluminator and electrophoresis setups, temperature controlled circulator with immersion stirrers suitable for inactivation studies at volumes up to 2 L per reactor, BSL level 2 fume hood, laminar hood, soil sampling equipment, Percival Scientific environmental chamber (model 1-35LLVL), custom-built rainfall simulator.

The Drexel Air Resources Research Laboratory (DARRL) is located in the Alumni Engineering Laboratory Building and contains state-of-the-art aerosol measurement instrumentation including a Soot Particle Aerosol Mass Spectrometer (Aerodyne Research Inc.), mini-Aerosol Mass Spectrometer, (Aerodyne Research Inc.), Scanning Electrical Mobility Sizer (Brechtel Manufacturing), Scanning Mobility Particle Sizer (TSI Inc.), Fast Mobility Particle Sizer (TSI Inc.), Centrifugal Particle Mass Analyzer (Cambustion Ltd.), GC-FID, ozone monitors, and other instrumentation.  These instruments are used for the detailed characterization of the properties of particles less than 1 micrometer in diameter including: chemical composition, size, density, and shape or morphology. 

In addition to the analytical instrumentation in DARRL, the laboratory houses several reaction chambers.  These chambers are used for controlled experiments meant to simulate chemical reactions that occur in the indoor and outdoor environments.  The reaction chambers vary in size from 15 L to 1 m3, and allow for a range of experimental conditions to be conducted in the laboratory.

Computer Equipment and Software

The Civil, Architectural, and Environmental Engineering Department at Drexel University has hardware and software capabilities for students to conduct research. The CAEE department operates a computer lab that is divided into two sections; one open access room, and a section dedicated to teaching. The current computer lab has 25 desktop computers that are recently updated to handle resource intensive GIS (Geographic Information Systems) and image processing software. There are a sufficient number of B&W and color laser printers that can be utilized for basic printing purposes.

Drexel University has site-licenses for a number of software, such as ESRITM ArcGIS 10, Visual Studio, SAP 2000, STAAD, Abaqus and MathworksTM Matlab. The Information Resources & Technology (IRT) department at Drexel University provides support (e.g., installation, maintenance and troubleshooting) to the abovementioned software.  It is currently supporting the lab by hosting a software image configuration that provides a series of commonly used software packages, such as MS Office and ADOBE Acrobat among others.  As a part of ESRI campus license (the primary maker of GIS applications, i.e. ArcGIS) the department has access to a suite of seated licenses for GIS software with necessary extensions (e.g., LIDAR Analyst) required for conducting research.  

Edmund D. Bossone Research Center

The Bossone Research Enterpirse Center includes 48 teaching laboratories, 37 lab support spaces, eight conference rooms, 77 offices and a 300-seat auditorium. The College of Engineering will occupy most of the building and will provide facilities for faculty and students from various departments in the University.  The Bossone Center is home to the Centralized Research Facilities (CRF), a collection of core facilities which contains resources for materials discovery and innovation, including structure, property characterization and device prototyping. Led by faculty and professional staff, the CRF serves a user base of more than 250 students, staff and faculty from across the University, and from its academic, national laboratory and industry partners in the Delaware Valley and beyond.

Machine Shop

Drexel University's College of Engineering offers a full-service fabrication Machine Shop on its University City campus. The facility has four full-time machinists with a combined industrial and academic experience of more than 100 years. The Shop is a multi-purpose machining facility capable of meeting all design needs. The Shop and its staff specialize in the research and academic environment, scientific instrumentation, biomedical devices, testing fixtures and fabrications of all sizes.

The EWT is offered several times per year, generally near the beginning and the end of every semester. Upcoming test dates are posted on the EWT registration portal, as well as on the information board in the Centre for Engineering in Society, near EV 2.249.

You must register in advance to take the test.

If you are a current Gina Cody School undergraduate student, you may register yourself on the EWT website. Use your ENCS account and password to log in and register. From outside Concordia, you must use VPN/MFA to access this app.

If you are not a current Gina Cody School undergraduate student, or if you any difficulties registering for the test, you should contact the test coordinator to register. Please see the test coordinator’s contact information on the lower right-hand side of this page.

Once you register for an EWT session, you must take the test. Unexcused absences count as failed attempts. If you have a valid reason for not attending the test after you have registered, you must contact the test coordinator before the date of the test to cancel your registration.

The results of the EWT are posted within one week of the test. You will find the results online in the registration portal, and also posted on the information board in the Centre for Engineering in Society, near EV 2.249.

If you pass the EWT, your results will be transmitted to Student Academic Services and you will be released to register for ENCS 282.

If you fail the EWT, you may attempt the test a second time, or you may choose to enroll in ENCS 272. After two failed attempts, students will be blocked from further registration into the EWT and will have to take ENCS 272 in order to fulfill the writing skills requirement.

Bring your student ID card to the test, as well as several pens. Please note that you may NOT bring the following into the EWT:

• Dictionaries
• Phones
• Calculators
• Other electronic devices

Originally Published MDDI January 2002

TEST SYSTEMS

Test System Engineering for Medical Devices: A Guide

Developing test systems for R&D through production requires a combination of preparedness and ongoing evaluation.

Tore Johnsen

A large number of medical device manufacturers use automated test and measurement systems during each stage of the product cycle, from R&D to production testing. These systems play an important role in improving the quality of products and helping speed them to market. Purchasing the test equipment is often less expensive than putting it into use; it can cost more to develop the software to run a system than to purchase the instrumentation. Many companies choose to outsource all or a portion of their test system development. Whether it's done internally or by an outside vendor, the critical factors for success remain the same: maintaining good communication between developer and client, following an efficient development process, selecting appropriate development tools, and recruiting people with the skills to do the job correctly.

This article provides a broad overview of test and measurement system development for the medical device industry. Included is a discussion of commonly used instrumentation and tools and an overview of the skills and practices necessary for successful test system development.

HOW AND WHY TEST SYSTEMS ARE USED

In the medical device manufacturing industry, test and measurement systems are used for a wide range of purposes. Some examples include those developed to:

• Demonstrate proof-of-concept prototypes for new equipment used to cauterize heart tissue.
• Research the shrinkage of new dental filling materials as they cure.
• Verify that such implantable devices as pacemakers and neurostimulators function according to specifications.
• Condition and test batteries for implantable devices.
• Simulate the pressing of buttons, turning of knobs, collecting of patient data, and other functions while monitoring and recording a blood-treatment machine's operation.

In order to carry out these tasks, the required systems must be capable of performing such functions as automatically controlling switches and relays, setting and studying temperatures, measuring or generating pulse widths and frequencies, setting and measuring voltages and currents, moving objects with motion control systems, using vision systems to detect misshaped or missing parts, and others. The medical device industry's need for test equipment and related sensors and technologies is as varied as the industry itself. No matter what the specific need, however, compliance with the quality system regulation is mandatory.

TEST SYSTEM TRENDS

Some trends have emerged as the popularity of automating test systems has grown. For instance, integrating test systems with a corporate database is becoming more common. A number of test stations can be networked, and data can be stored in the corporate database. From that database, important statistical process data can be gathered and analyzed.

Additionally, manufacturers are growing more interested in using standard Web browsers to view data and in remotely controlling their test systems. Data are routinely passed between test applications and standard office applications such as Microsoft Excel and Word. Both the complexity and number of technologies being incorporated into the systems are growing, and, consequently, so are the demands on systems developers and project managers. Test system developers are using more-efficient software development and project management methodologies to meet these increased demands.

Standardizing specific test system development tools and instrumentation is another increasingly popular way to keep costs relatively low. Using the same development tools from R&D to production has obvious benefits: it reduces training costs, allows for technology reuse, and makes it easier to shift employees from one area to another, depending on demand. If executed with diligence, maintaining consistency also facilitates the creation of a library of reusable code. Standardizing makes compliance with quality systems requirements easier, too.

TEST SYSTEM INSTRUMENTATION

A typical test system is created in one of two ways. It can be built around a PC using a plug-in data acquisition board (DAQ), serial port instruments, or general-purpose interface bus (GPIB). Alternatively, it may be built around a chassis with an embedded computer and various plug-in modules such as switches, multimeters, and oscilloscopes (see Figure 1).

The existing member of the latter family of instrumentation systems is the PXI chassis, which uses the standard PCI-bus to pass data between the plug-in modules and the embedded computer. This technology offers high data acquisition speeds at a relatively low cost. Originally invented by National Instruments (Austin, TX), it is now an open standard supported by a large number of instrument vendors. (For additonal information on this system, visit http://www.pxisa.org).

Manufacturers should keep in mind the several variations that exist on these schemes. To select the optimal instrumentation for any given project, a company must consider both the cost and performance requirements of the project, and the need for compliance with internal and external standards.

An important system-selection criterion is the availability of standardized ready-made software modules (i.e., instrument drivers) that can communicate with the instruments. Because the major instrument vendors are currently involved in a significant instrument-driver standardization effort, it makes sense for companies to check the availability of compliant instrument drivers before purchasing an instrument. The Interchangeable Virtual Instruments Foundation (http://www.ivifoundation.org) works on defining software standards for instrument interchangeability. The foundation's goal is to facilitate swapping of instruments with similar capabilities without requiring software changes. Creating a custom instrument driver can take from days to weeks depending on the complexity of the instrument; if it's not done correctly, future maintenance and replacement of instruments might be more difficult than need be.

DEVELOPMENT TOOLS

Development tools designed specifically for PC-based test system development have been in existence for more than a decade. Specialized graphical languages for drawing, rather than writing, programs are useful. One such product from a major instrumentation vendor is LabVIEW (National Instruments). Developing everything from scratch in C is probably not a good idea—unless there is plenty of time and money to spare.

As alternatives to specialized graphical languages, several add-on packages exist that can make a text-based programming language more productive for test system development. For example, one can buy add-ons for both Visual Basics and Microsoft Visual C++. If one's preference is C, for instance, but the benefits of a large library of ready-made code for instrumentation, analysis, and presentation are desired, LabWindows/CVI from National Instruments is a tool to consider.

If what's needed is development of an advanced system to make automated logical decisions about what test to run next and to perform loops and iterations of tests—and store the results in a database—a test executive that can work with the test code is a wise option. Figure 2 shows an example of test-executive architecture. The test system developer writes the test modules and then uses the test executive's built-in functionality to sequence the tests, pass data in and out of databases, generate reports, and make sure all test code and related documentation is placed under revision control (i.e., configuration management).

Although this option still requires significant effort to develop the test modules and define the test sequences, using a standard test executive is, in many cases, far more cost-effective than making one from scratch. This is especially true for such large, ongoing projects as design verification testing and production testing, which require regular modifications of tests and test sequences.

THE NECESSARY SKILLS

A test system development project requires a multitude of skills to achieve success, including project management skills and good communication to keep the project on track and to ensure that all stakeholders' needs and expectations are addressed. Understanding and practicing good software development methodologies are also needed to ensure that the software that is built will actually meet the user's requirements. Test system development also requires that engineers have a thorough understanding of software design techniques to ensure that the software is both functional and maintainable, and an understanding of hardware and electronics to design the instrumentation and data acquisition portions of the system.

Before a test system can be put into production, it needs to be tested and validated. This means that the development team also needs the expertise to put together a test plan and to execute and document the results in a report. The engineers who built the system are not necessarily the best people to test it, so additional human resources are often needed for testing. Finally, because documents are created during the development process, documentation skills are also necessary.

When one considers that the typical project team for a midsize test system consists of two to four developers, one realizes there are more major skills required than there are team members; therefore, one of the challenges is to locate individuals with sufficiently broad skills and abilities to supply both technical and managerial leadership. To ease this burden, make the tasks less daunting, and increase the chances of project success, defining a development process is key. If the test system is used for regulated activities, such as production testing of medical devices, then the test system itself is subject to the quality system regulation and a defined development process is not only desirable, it's mandatory.

THE BENEFITS OF COLLABORATION

Outsourced projects are most successful when the developers and the clients collaborate. Keeping the client involved is the most efficient way of making sure that the system meets the client's needs. It also helps avert surprises—at either end—down the road. Collaboration requires honest and direct communication of issues, successes, and problems as they occur.

Miscommunication sometimes happens even with good collaboration. While it is important to keep the communication channels open so the developers and their clients can discuss issues without too much bureaucracy, it can be hard to keep track of who said what if too many parallel communication channels exist. And when engineers on both sides have ideas of features they would like to add to a particular system, controlling feature creep can become difficult.

Designating a single point of contact for discussing both a project's scope and its content is recommended, and making sure new solutions are reviewed before being accepted can also prevent problems. Instituting a change-control procedure is yet another important step to minimizing unnecessary changes.

THE PROJECT PROCESS

The goal for any project is to add only as much process overhead as is absolutely necessary to satisfy the objectives. When a process must be added because regulations mandate it, the involved parties should keep in mind that the process isn't being instituted merely to satisfy FDA or other agencies; it's being done to build better and safer products. Structure and process improvements can have a significant positive impact on the quality of the finished test system.

The Software Engineering Institute has defined the following key process areas for level 2 ("Repeatable") of the capability maturity model: requirements management, project planning, project tracking and oversight, configuration management, quality assurance, and subcontractor management.¹ The foundations for a project's success are good requirements development and good project planning; if the requirements aren't right, or if a company can't determine how to get the project done, then the project is essentially doomed. What follows is a description of the progression of a few types of test system development projects as well as a discussion of requirements development.

Phases of Test System Development. Whether a formal documented development process is followed or not, there are a few major tasks (i.e., project phases) that must be addressed: requirements development, project planning, design, construction, testing, and release. Should this particular order of tasks always be followed? Probably not. During the 1980s the software industry saw a number of large projects go significantly over budget, become significantly delayed, or be cancelled because they were inherently flawed. One cause of these problems was companies' strict adherence to the waterfall life-cycle model (see Figure 3). In this type of life cycle, the project goes through each phase in sequence, and the phases are completed one at a time. The waterfall model presumes that the requirements development phase results in nearly perfect requirements, the design phase results in a nearly perfect design, and so forth.

Unfortunately, projects are normally less predictable and run less smoothly than the waterfall model assumes. For example, a company doesn't always know enough at the beginning of a project to write a complete software requirements document. The sequence of actions necessary for project success depends to a large extent on the nature of the project. Because every project is unique, those involved must analyze the project throughout its phases and adapt the process accordingly.

Keeping that in mind, software companies have done much to Boost software development methods since the 1980s. Today, one can find descriptions of a number of life-cycle models useful for different project characteristics. Choosing the appropriate life-cycle model depends on the nature of the project, how much is known at the start of the project, and whether the project will be developed and installed in stages or all at once. Of course, mixing and matching ideas from different life-cycle models can be an effective strategy as well. Even if a company has decided upon and made standard a particular life-cycle model, small modifications should be made to that model when a particular project necessitates it. The trick is to identify high-risk items and perform risk-reducing activities at the start of the project.

Test System Characteristics. The test systems used in the three device development stages—R&D, design verification, and production—each have their own characteristics.

R&D Systems. R&D test systems range in development time from a few days to many months. Scientists use the systems to explore new ideas; R&D test systems are also used to perform measurements and analyses not possible with off-the-shelf equipment and to build proof-of-concept medical equipment. Others are used by physicians for medical research. Vastly varied in both scope and technologies, most R&D test systems have one thing in common: the need for continuous modification and development. As the research progresses, the scientist learns more, generates more ideas, and might decide to incorporate a new functionality or new algorithms, or even try a different approach. Clearly the waterfall life-cycle model doesn't fit such developments. With R&D test systems, one doesn't know what the final product will look like at the start of the project. In fact, there might not be a final product at all, just a series of development cycles that terminate when that particular research project is over and the system is no longer needed.

Assuming that a reasonable idea of the scope of the R&D project is known at the start, one possible life-cycle model to follow is the evolutionary-delivery model (see Figure 4). This model includes the following steps: defining the overall concept, performing a preliminary requirements analysis, designing the architecture and system core, and developing the system core. Then the project progresses through a series of iterations during which a preliminary version is developed and delivered, the client (i.e., the researcher) requests modifications, a new version is developed and delivered, et cetera, until revisions are no longer needed.

Of course, it's wise to try to pinpoint potential changes at the beginning of the test system development project so that the software architecture can be designed to handle the changes that might come later on.

Design Verification Test Systems. There often is a blurry line between R&D and design verification test (DVT) systems. In the final stage of DVT system usage, the output is verification and a report stating that the medical device performs according to its specifications. Before that stage is reached, however, it is not uncommon to encounter several DVT cycles, each delivering valuable performance data back to the device's designers, and each resulting in modifications either to the device's design or to its manufacturing process.

It may be desirable to use the DVT system to test parts of the device or portions of its functionality as soon as preliminary prototypes are available, but it may not always be possible to have the complete test system ready for such applications. In these cases, the staged-delivery life-cycle model (see Figure 5) may be the best choice. According to this model, test system development progresses through requirements analysis and architectural design, and then is followed by several stages. These subsequent stages include detailed design, construction, debugging, testing, and release. The test system can be delivered in stages, with critical tests made available early.

Production Test Systems. A production test system needs to be validated according to an established protocol.² Such a test system is therefore developed and validated using a well-defined process, and the system can normally be well-defined in a requirements specification early on. There is still, however, a long list of possible risk factors that, if realized, can have a serious negative impact on the project if a strict waterfall development life cycle is followed. Research has shown that it costs much more to correct a faulty or missing requirement after the system is complete than it does to correct a problem during the requirements development stage.

A risk-reduced waterfall life cycle might be an appropriate model to follow when developing a production test system. In this life-cycle model, main system development is preceded by an analysis of risks and a performance of risk-reducing activities, such as prototyping user interfaces, verifying that unfamiliar test instruments perform correctly, prototyping a tricky software architecture, and so forth. Iterations are then performed on these activities until risk is reduced to an acceptable level. Thereafter, the standard waterfall life cycle is followed for the rest of the project—unless it is discovered that some new risks need attention.

Requirements Development. As the aforementioned life-cycle models show, requirements development directly influences all subsequent activities. It's important to remember that the requirements document also directly influences the testing effort. Writing and executing a good test plan are only possible when a requirements document exists that clearly explains what the system is supposed to do.

Developing a software or system requirements document is important, but there is no one perfect way to do it. Depending on the nature of the project, the life-cycle model selected, and how well the project is defined at its early stages, the requirements document might use a standardized template and be fairly complete, it might be a preliminary requirements document, or it might simply take the form of an e-mail sent to the client for review. No matter how it's done, putting the requirements in writing improves the probability that both parties have the same understanding of the project.

Test system developers also are well advised to create a user-interface prototype and prototypes of tangible outputs (e.g., printed reports, files, Web pages) from the system. These might take the form of simple sketches on paper or real software prototypes. The purpose of the user-interface prototype is to make sure the software maintains the correct functionality. Often, the first time clients see a user interface, they remember features they forgot to tell the developer were needed, and they realize that the system would be far more valuable if greater functionality were added. Creating a user-interface prototype is perhaps the most efficient method for discovering flawed or missing functional requirements. Both parties will want this discovery made during the requirements development phase, not upon demonstration of the final product.

To the greatest extent possible, developers should identify any items that are potential showstoppers, such as requirements that push technology limits or the limits of the team's abilities. Identifying such problems might require some preliminary hardware design to ensure the system actually can be built as specified. High-risk items should be prototyped and developers should try to identify ways to eliminate the need for requirements that push the limits. Waiting until the final testing stage to find out that some requirements cannot be met is not a good idea. Even waiting until after the requirements are signed off to find that some cannot be met is unpleasant—especially if all it would have taken to prevent the problem was a few hours' research.

For outsourced development projects it is essential that the test system developer get feedback from the client and iterate as needed until an agreement is reached and, in some cases, the requirements are signed off. While performing the activities described above, the developer also should review any solutions suggested or mandated by the client. For instance, if the client says it already has the hardware and only wants the test developer to provide the software, the first thing the developer should do is request a complete circuit diagram of that client's hardware solution and carefully explain why it's necessary to fully understand the client's hardware in order to build a good software system. Flaws in the test instrumentation design are very costly to fix after the test system is built, yet it costs comparably very little to review the design ahead of time. Of course, an in-house test system developer also should evaluate the hardware design carefully before starting the software design.

Project Review Timing. It's likely that outside developers who get this far have dealt solely with the client's project team. If the project is large, however, it is not uncommon for the client to bring more people into the picture and conduct a project review after the system is complete. Some of the newcomers will have insights and desires that would result in changes—sometimes expensive ones.

If possible, this type of situation should be avoided. The system developer should insist on a project review by all affected parties before the requirements stage is concluded. It is not enough to just send around the software requirements specifications; people are often too busy with other projects to really go through them as meticulously as they should. A better strategy is to bring everybody together and show them the user-interface prototypes, the report prototypes, and any other important components of the project. Representatives of the end-users should be present as well. Although they should have been consulted during the requirements development process, the end-users are still likely to contribute valuable insights during the review.

By now it should be evident that there is more to requirements gathering than just writing a requirements document and getting it signed off. If the system doesn't work to the client's satisfaction at delivery, then it doesn't matter who is to blame. The project will be remembered by both parties as a painful experience with no winners.

PROJECT PLANNING

Every project needs a plan. The first step in project planning is to define the deliverables, then to create a work breakdown structure (WBS) hierarchy of all the project's required tasks. The WBS is then used to develop a timeline, assign resources, and develop a task list with milestones. A good project plan will also clearly define roles, responsibilities, communication channels, and progress-report mechanisms. It certainly helps to have some background or training in project management in order to plan and control the execution of the project. Some basic project management training is recommended for anyone in charge of a test system development effort. Seminars and classes in project management based on the Project Management Institute's standards are offered worldwide.

CONCLUSION

Successful test system development requires attention to both process and technology. Both clients and developers need to understand and appreciate good software engineering practices. Collaboration and communication are critical for success. Clearly defining roles and responsibilities, using efficient development processes and tools, and handling project risks early on permits problems to be handled at a stage when their effect on cost and schedule will be minimal.

REFERENCES

1. Capability Maturity Model for Software, Version 1.1, Technical Report CMU/SEI-93-TR-024-ESC-TR-93-177 (Carnegie Mellon University, Pittsburg, PA: The Software Engineering Institute/February 1993).

2. Medical Device Quality Systems Manual: A Small Entity Compliance Guide, (Rockville, MD: FDA, 1997).

BIBLIOGRAPHY

A Guide to the Project Management Body of Knowledge. Newton Square, PA: Project Management Institute Standards Committee, 2000.

McConnell, Steve. Rapid Development. Redmond, WA: Microsoft Press, 1996.

McConnell, Steve. Software Project Survival Guide. Redmond, WA: Microsoft Press, 1998.

Wiegers, Karl E. Software Requirements. Redmond, WA: Microsoft Press, 1999.

Tore Johnsen is technical director and en-gineering manager at Computer Solutions Integrators & Products in Woodbury, MN.

Figures 3 and 4 adapted from Rapid Development, with permission of the author, Steven C. McConnell.
Figure 5 adapted from Rapid Development, with permission of the author.

Copyright ©2002 Medical Device & Diagnostic Industry

For details regarding the items below, please review the Admission Application Instructions.

Graduate Admission Application

Applicants may only apply to one program. All documents submitted by you or on your behalf in support of this application for admission to Drexel University become the property of the University, and will under no circumstances be released to you or any other party. Please note, an application fee of $65 U.S. is required.

Apply Now

Transcripts

Official transcripts must be sent directly to Drexel from all of the colleges and universities that you have attended. Please note that transcripts are required regardless of number of credits taken or if the credits were transferred to another school. An admission decision may be delayed if you do not send transcripts from all colleges and universities that you have attended.

Transcripts must show course-by-course grades and degree conferrals. If your school does not notate degree conferrals on the official transcripts, you must provide copies of any graduate or degree certificates.

If your school issues only one transcript for life, you are required to have a course-by-course evaluation completed by an approved transcript evaluation agency.

Precise, word-for-word English translations of all non-English language documents are required along with official documents in the original language. All translations must be completed by the issuing institution or an ATA-certified translator. Visit the American Translators Association website to search for an ATA-certified translator.

International students: If you have already graduated from your previous institution at the time of your application, please email your graduation certificate(s) attached as PDF or Microsoft Word documents to enroll@drexel.edu. Note: Any international applicant, regardless of program, may be required to provide a transcript evaluation should the Admissions Committee determine it is needed for the application to be reviewed. If your transcripts are selected for a course-by-course evaluation by an external agency, you will be responsible for supplying all necessary documentation and paying all necessary fees to have your transcripts evaluated.

If you have questions regarding what documents you must submit to fulfill the transcript requirements, contact the Office of Graduate Admissions.

Standardized Test Scores

GRE test scores are not required for MS applicants and waived for PhD applicants through the fall 2022 application deadline. If you have already taken the GRE exam, we encourage you to submit your scores. They will be included in your application review and are a consideration in the evaluation for eligible scholarships and fellowship awards.

Test of English as a Foreign Language

International applicants are required to demonstrate English language proficiency by submitting scores from the Test of English as a Foreign Language (TOEFL minimum scores: 90/577/233), the International English Language Testing System (IELTS minimum Overall Band Score: 6.5), the Pearson Test of English (PTE minimum score: 61), or the Duolingo Test (Duolingo minimum score: 110) unless they meet the criteria for a waiver.

Essay

Please write approximately 500 words explaining your reasons for pursuing a degree from Drexel; your short-term and long-term career plans; and how your background, experience, interest, and/or values, when combined with a Drexel degree, will enable you to pursue these goals successfully.

Submit your essay with your application or through the Discover Drexel portal after you submit your application.

Résumé

Upload your résumé as part of your admission application or through the Discover Drexel Portal after you submit your application.

Letters of Recommendation

Two letters of recommendation are required. To electronically request recommendations, you must list your recommenders and their contact information on your application. We advise that you follow up with your recommenders to ensure they received your recommendation request — they may need to check their junk mail folder. Additionally, it is your responsibility to confirm that your recommenders will submit letters by your application deadline and follow up with recommenders who have not completed their recommendations.

Request recommendations with your application or through the Discover Drexel portal after you submit your application.

Alternatively, you may submit your recommendation letters by mail. Letters must include the address, phone number, and signature of the recommender. The recommendation envelope must be submitted unopened to the Office of Graduate Admissions.

Mailing Addresses

Most teams are somewhere on the path to DevOps maturity, with just 11% saying they’ve implemented full automation in DevOps. This means that despite being around for almost two decades, most organizations are still figuring out what full DevOps adoption looks like for their teams. However, after years of disruption, rising customer expectations for digital experiences, and economic turmoil, C-suite patience for gradual transitions is limiting, pushing software teams to overcome long-term DevOps hurdles and prove ROI. 

Quality engineering is the practice of including quality testing throughout the development lifecycle, with the purpose of delivering a positive user experience that will help you satisfy, retain, and acquire new customers. It is emerging as a complement to DevOps that helps teams overcome common challenges to transformation. By focusing their efforts on quality engineering, development leaders can help their organizations finally achieve DevOps success. 

Software Testing Transformation Supports Customer Happiness During DevOps Transition

Let’s face it: change, even positive change, is hard. Many software companies find themselves trapped between the need to modernize their development pipelines and the concern that the transition will cause too much disruption for their customers. The latter often wins out as most businesses now compete on their customer experience: 75% of American consumers say it plays a major role in their purchasing decisions, and 32% say they’ll leave a brand after just one bad interaction. This highly competitive environment creates an understandable aversion to change that slows many DevOps adoptions.

Building the DevOps journey around technologies and processes that support better customer experiences helps software development teams overcome the inevitable headaches that occur during transitional phases and ensures customers don’t suffer the consequences. According to mabl’s 2021 Testing in DevOps Report, 80% of teams with high test coverage reported high customer satisfaction. This number is impressive on its own, but even more striking when compared to organizations with low test coverage: just 30% of low test coverage teams reported high customer happiness. 

Rather than abandon DevOps adoption in the pilot stage, automating software testing is a low risk, high reward place to embrace automated, collaborative DevOps pipelines without risking the user experience. 

Streamlining Collaboration in DevOps Pipelines

DevOps seeks to support faster, more dynamic development teams by building a shared workflow that emphasizes collaboration. When product owners, developers, and quality professionals can easily work together, they’re able to easily hand off issues so that defects are addressed quickly. It’s no surprise, then, that the further teams were in the DevOps adoption process, the better they felt about collaboration between teams. 

Starting the DevOps journey by evaluating points of collaboration like the handoff between quality teams and engineering addresses a major challenge to DevOps adoption: slow processes and a reluctance to change. Streamlining these essential functions not only helps individual team members see the value in DevOps adoption, it also reduces the likelihood that customers will have a bad experience as the result of a software defect that escaped into production. With defects easier to manage, DevOps teams can focus on improving the product, adding new features, and making the overall customer experience better. 

Closing the DevOps Loop

With just 11% of organizations saying that they’ve reached fully automated pipelines, it’s clear that momentum is still building for DevOps adoption. But while DevOps is still a priority for many software development teams, there are still serious obstacles on the road to success. And as the world enters a new period of highly competitive market conditions, the teams that can successfully modernize their pipelines for iterative, quality-centric development will be best positioned to succeed. The time for DevOps experimentation is over – it’s time for DevOps success. 

Improving software testing with test automation and better cross-functional collaboration processes is an underestimated – and undervalued – avenue to DevOps maturity that can help software organizations finally realize their goals. By emphasizing quality engineering metrics like test coverage and how well quality and engineering teams can collaborate, DevOps leaders will be better prepared to showcase business value and tackle the cultural shifts that continue to inhibit DevOps maturity.

Content provided by mabl.