I. Introduction
Users of any modern software system most probably expect their software system to perform effectively, be usable and function reliably. Users define a list of requirements for a software system before its development process starts. These requirements have to be assessed by software developers throughout their development process. Software developers then select a software development process model like spiral, waterfall and v-models for developing that software system.
Like all real-world, man-made products, software systems also must be tested in order to meet user requirements and to ensure the quality of a software system. A software system without a good quality cannot live long or see the light of a day. The main reason for bad or low quality of a software system is software errors.
Software errors are usually syntactic, semantics errors, software bugs, faults or failures in a software system. A software error is a semantic or syntactic incorrect part of a programming operator or an operand. A software bug is a technical error in a software system. A software fault is a functional error or wrong part of software system. A software failure is a failed part of a software system which does not work or give a result as user expects.
Software testing is a process of testing and verifying the developed or planned software system if it meets and satisfies user requirements. This paper focuses on the topic of software testing which is the main and integral part of all software development process models. By using the principles and activities of software testing in the software development process, developers aim at making their software more reliable by increasing functional correctness and stability of the software system.
Considering all aforementioned discussions, this talk aims at providing a complete survey on the topic of software testing. The topics covered in this work might for example be used as a structure for a lecture on software testing.
In the following, the main goals of the problem area are motivated in Section II. In Section III, definitions of the most important parts of software testing are portrayed. Section IV gives some ideas how to evaluate software testing in the field of software quality assurance. This paper ends up in Section V by drawing some conclusions and outline for future work.
II. Motivation
Software systems have long been used everywhere in our daily life and society. They have to satisfy the quality requirements. Particularly, software systems must have high quality in critical fields such as monetary transaction industries — banking and financial systems, human life-related areas, aviation, military, security and other areas. Running a software system with bad quality in such critical infrastructures might cost human life or a large amount of financial damage. For example, a small error in a software for banking and financial system might result in a huge amount of financial loss by hacker attacks during the financial transactions.
During the process of software development, software developers might make mistakes in the source code of a software system which cause very serious problems later. Therefore, software testing is necessary and main aspect in ensuring the quality of a software system and avoid possible errors, bugs, faults and failures in a software systems. Some of those mistakes are unimportant, but some of them are very expensive and dangerous. Since software systems are running in all the fields of society such as bank, finance, administration, military, communal system and daily life, all the aspect of a software system and every single piece of source code must be tested and verified before things go wrong and costs human life, catastrophe, financial loss or others.
If a software system is not tested, no one can guarantee the full functionality, reliability and security of that software and if it satisfies all the requirements of users. A non-tested software system can even bring a lot of stress, financial loss and ineffectiveness to its users performing very low quality.
Of course, software quality can be increased by test coverage. In order to develop software systems with good quality, software developers should have a lot of experience and knowledge on software testing. Also, a developed software system requires a certain level of knowledge of software stakeholders. Because, a user has to be capable of verifying the percentage of test coverage of his software which ensures reliability of that product and the software was developed and functions as intended.
III. Software Testing
3.1 Objectives of software testing
Test cases written on implementation level aim at making sure that the implementation of a system fulfills a certain level of software quality and stability. But, there are more aspects and objectives to software testing that aim at making a system tested and covered from top to bottom including the part that is actually used by users. These testing objectives are described in this section.
A.Correctness
One of the main purposes of software testing is checking whether a system behaves correctly in the meaning of not producing wrong results or behaving unexpectedly. To this end, software systems are the subject to be tested on different levels. The correctness tests are applied to the system more on implementation level, e.g. a system cannot simply be checked for correctness in terms of the following objectives.
B.User Interface
The process of testing user interfaces (UI) of software systems is a very nontechnical step in the testing process. The automated or tool-supported for testing a UI is not appropriate. Hence, a tester is supposed to look at the following aspects:
Standardization and Guidelines: Depending on the use case of the system or the operation system it will be used on, there may exist standards on how end users expect the UI to be layouted and designed.
Intuitive: UIs should be intuitive for the end user which may also be covered by guidelines and standardization.
Consistency: UIs should be consistent throughout the system so that the UI never behaves differently than the user would expect from a previous action.
C.Usability
Another approach for testing a user interface is to create usability tests. In contrast to the UI testing process, usability testing uses studies of the behavior, expectations and feedback of real users of the software system. This process can be aligned to the general testing process. Test cases have to be derived, e.g. a specific use case of the system should be executed by surveyed users and the results will be reported. Users have to be selected for the survey and their results have to be evaluated. The results can be used to improve the usability of the software and the next iteration of usability testing can begin.
D.Performance
Non-functional requirements for a software might often include performance requirements such as response times. Performance Testing aims at testing whether such requirements are fulfilled. Performance tests might however not always be accurate enough, due to their nature of being executed in a test environment, so that a general fulfillment of a performance requirement can be guaranteed in all environments.
3.2 Software testing management
Test management comprises not only classical methods of project and risk management but also knowledge about suitable test methods. Test management helps to select and implement appropriate measures to ensure that a defined basic product quality will be achieved. So the test management includes a test process for better structure.
A.Test Process
A detailed test procedure is necessary to integrate testing into the development process. The software testing consists of seven steps, which are described below.
— Organization for Testing: In this step the test scope should be defined. It means the performed testing type should be determined. The organization of the test team occurs in this stage, too. In this step it is also important to assess the development plan and status. This helps to build the test plan.
— Developing the Test Plan: Identification of the test risks and writing the test plan belongs to the development of a test plan. It is important to follow the same pattern as any software planning process. The structure is the same, but this plan focuses on the degree of the tester perceived risks.
— Verification Testing: During this step the testers have to determine that the requirements are accurate and complete, and that they are not in conflict to each other. The second important point here is to concern that the software design will achieve the objectives of the project as well as it will be effective and efficient. The software constructions have to be tested here, too.
— Validation Testing: At this step the validation testing should be performed. This involves the testing of code in a dynamic state. The results of these tests should be recorded.
— Analyzing and Reporting Test Results: Analyzing and reporting of test results belongs to the fifth step of test planning. The objectives of this step are to determine what the team has learned from testing and to inform appropriate individuals.
— Acceptance and Operational Testing: In this step one should perform acceptance, software installation and software changes testing.
— Post-Implementation Analysis: The objective of this step is to determine whether testing was performed effectively and what changes could be made, if not.
3.3 Classification of tests
Test levels are a group of test activities that are organized and managed together. A test level is linked to the responsibilities in a project. Examples of test levels are component tests, integration tests, system tests and acceptance tests.
Testing techniques are procedures to derive and/or select test cases, i.e Black Box tests and White Box tests.
Test objectives are a group of test activities aimed at testing a component or system focused on a specific test objective, i.e. functional test, usability test, regression test, etc. A test type may take place on one or more test levels or test phases.
A.Unit Testing
A level of the software testing process where units, components, functions or methods are tested to see if they perform as designed. In unit testing programmers test various program units, such as classes, procedures or functions until they satisfy a set of precise requirements.
B.Integration Testing
Testing performed to expose defects in the interfaces and in the interactions between integrated components or systems. It is performed directly after the unit testing and it is done with the collaboration of software developers and integration test engineers.
C.System Testing
The process of testing an integrated system is to verify that it meets specified requirements.
D.Acceptance Testing
Acceptance testing is a formal testing process with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether to accept the system.
3.4 Testing teams
The test activities need to be managed by people with a good understanding of the testing techniques and processes. The feedback derived from analyses of measurement data needs to be used to help with various management decisions, such as a product release, and to help quality improvement. Test managers are involved in these activities. Testers and testing teams can be organized into various different structures, but basically following either a horizontal or a vertical model:
A vertical model would organize around a product, where dedicated people perform one or more testing tasks for the product. For example, one or more teams can perform all the different types of testing for the product, from unit testing up to acceptance testing.
A horizontal model is used in some large organizations so that a testing team only performs one kind of testing for many different products within the organization. For example, different products may share the same system testing team.
3.5 Test automation
We conduct test automation with intent to automate some manual objectives by using the suitable software tools. The need of test automation is strong, because fully manual testing from beginning to end can facilitate boredom and human-errors. On the other hand, long standing theoretical results tell us that fully automated testing is impossible. The goal of using test automation to save people of tedious and repetitive tasks and to improve overall testing productivity is to first examine what is possible, feasible, and economical, and then to set the right expectations and aims. Various issues related to test automation include:
specific needs and potential for automation;
selection of existing tools, if available;
possibility and cost of constructing specific test automation tools;
availability of user training for these tools and time/effort needed;
overall cost, including costs for tool acquisition, support, training, and usage;
impact on resource, schedule and project management.
3.6 Tools
One important tool for software testing, especially in the domain of unit, integration and acceptance tests, are frameworks used to test a software system on different levels of implementation. JUnit is one of the available unit-testing frameworks for the Java Programming Language. It allows the repeated execution of Java-written test cases against a Java-written software system or parts of it. It can also be used for integration tests. Other frameworks are specifically designed for acceptance testing, e.g. Concordion or Selenium. With such frameworks at hand, developers and testers are able to repeatedly test software systems in short intervals.
IV. Software Testing for Software Quality
As long as software testing is a key foundation in ensuring the quality of a developed software system, it needs to be investigated in software quality assurance both software producers and software customers. Usually, both software development companies and their customers do not pay much attention to the internal test coverage of their software system's source code, only consider limited part of user interface and functionality of their software. In fact, if there is a generic tool which investigated test coverage and elicit detailed knowledge about software systems quality based on that test coverage, it would be quick essential for both software developers and software customers. Such generic quality control framework would also quite useful for patent offices to test and verify produced software systems and grant software developers to software license and authority. The authors of this paper consider to build a generic tool/framework for monitoring the quality of software systems based on their test coverage and classify the advantages of such framework as follows:
— For software developers/companies: Software developers or companies usually use software development, project management and continues integration system for developing their software products. But, they usually do not pay much attention to the test coverage rating and inner quality of their source code, especially young developers in the developing countries. A generic software quality monitoring system will help them to develop software systems with high quality by testing so that they can constantly keep an eye and monitor their software system.
— For software customers: A generic software quality monitoring system is very important for software customers and stakeholders. Because, software customers and stakeholders are not usually expert or do not have enough knowledge on programming, technology and software development and they always intend to know what they are paying for and see how high the quality of their software system they are buying. In such situations, they can easily use the generic software quality monitoring system in order to verify and check the quality of software system they are buying in the software market.
— For software patent offices: In every country in the world, there are patent office which train with granting software developers to have license and authority over the software systems they developed. In this case, the patent offices are usually in difficulty with identifying and recognizing the quality rating and test coverage percentage of software systems. In such situations, they feel a need for a generic software quality monitoring system to know all aspects of a software system so that they simply can push a software system into the quality monitoring system and review the quality aspects in a dashboard.
V. Conclusion
Software testing is a very important and integral part of the programming process. Various tests have to be conducted for every created program. This helps to improve the quality of any software system in general that play a critical role. Software testing ensures that the completed software shows high performance and responsibility. High test coverage is the good mark for software developer and the good motivation for customer to buy software products, because it ensures that developer receive his money and the customer gets the qualitative product. Software testing allows us to prevent many unwanted and unexpected problems, that can be small like an incorrect transaction or huge like an air catastrophe.
Early design decisions of the generic software quality monitoring systems are already made and initial design is finished. The development process of the framework is in process nowadays. The authors of this paper intend to introduce early release of the framework in early summer of this year.
References:
- A. Zeller, Why Programs Fail: A Guide to Systematic Debugging. San Francisco: Morgan Kaufmann Publishers Inc., 2005.
- I. C. Society, 610.12–1990 — IEEE Standard Glossary of Software Engineering Terminology. IEEE Computer Society, 1990.
- N. Fenton, Software Metrics: a Rigorous Approach. Chapman & Hall, 1991.
- I. C. Society, IEEE 829–2008 IEEE Standard for Software and System Test Documentation. IEEE Computer Society, 2008.
- I. S. T. Q. Board, Standard Glossary of Terms used in Software Testing. International Software Testing Qualifications Board, 2014.
- ISO/IEC, ISO/IEC 9126. Software engineering — Product quality. ISO/IEC, 2001.
- I. technical committee (TC) ISO/TC, ISO 9000 — Quality management. ISO technical committee (TC) ISO/TC, 2008.
- G. J. Myers, T. Badgett, C. Sandler, The Art of Software Testing. John Wiley & Sons, Inc., 2012.
- K. Naik and P. Tripathy, SOFTWARE TESTING AND QUALITY ASSURANCE Theory and Practice. John Wiley and Sons, 2008.
- P. Ammann and J. Offutt, Introduction to Software Testing. Cambridge University Press,08.
- R. Patton, Software Testing. Sams Publishing, 2001.
- W. E. Perry, Effective Methods for Software Testing. Wiley Publishing, 2006.
- A. Spillner, T. Linz, T. Rossner, M. Winter: Test Management: A Study Guide for the Certified Test Exam ISTQB Advanced Level. Rocky Nook, 2007.
- R. Patton, Software testing. Indianapolis, Ind.: Sams, 2001.
