Security systems of modern operating systems



As operating systems are enhancing more and more today, computer users can feel free of danger as long as security systems are robust and under control

Keywords: security level, E-commerce, Mandatory Access Control (MAC), commercial operatingsystems, cyber-space threats.

Introduction. Every modern computer system, from network servers, workstation desktops, to laptops and hand-held devices, has a core piece of software, called kernel or operating system, executed on the top of a bare machine of hardware that allocates the basic resources of the system (e.g., CPU, memory, device driver, communication port, etc), and supervises the execution of all applications within the system. Some popular commercial and Open Source operating systems are Microsoft Windows, different flavors of Unix (BSD, AIX, HP-UX, Solaris, etc), Mac OS, and Linux. Because of the crucial role of the operating system in the operation of any computer systems, the security (or lack of security) of an operation system will have fundamental impacts to the overall security of a computer system, including the security of all applications running within the system. A compromise of the underneath operating system will certainly expose danger to any application running in the system. Lack of proper control and containment of execution of individual applications in an operating system may lead to attack or break-in from one application to other applications.

Based on the “Trusted Computer System Evaluation Criteria” of US government [1], the security level of most commercially available operating systems are no higher than C2 class, which requires Discretionary Access Control (DAC) protection at a per user granularity. Although this level of protection provides safeguard of certain extent among different applications in a multi-tasking, timesharing environment that is typical for current mainstream operating systems, no mechanisms are supported by operating systems in this class to enforce strict security policies of individual applications. As a result, in a C2 class operating system the security of applications and users are responsible for their own fates. With the ever-growing connectivity and E-commerce through the Internet, application security is an ultimate goal for millions of merchants and consumers who turn their business and service electronic and to the public world of cyberspace. On the other hand, efforts to achieve total security of such systems continue to be based on the flawed promise that adequate security can be achieved in applications with the current security mechanisms of mainstream operating system [2]. The reality is that secure applications demand secure operating systems, and tackling application compromises at the OS level by kernel-enforced controls should probably be considered as an attractive and effective approach. In order to raise the security level of operating systems to next class — B class, the requirement of Mandatory Access Control (MAC) is a necessity. A typical MAC architecture needs the ability to enforce an administratively set security policy over all subjects and objects (users, processes, memory, files, devices, ports, etc) in the system, basing decisions on labels containing a variety of security-relevant information. MAC provides strong separation (or containment) of applications that permits the safe execution of untrustworthy applications, and enables critical processing pipelines (trusted path) to be established and guaranteed. Therefore, it offers critical support for application security by protecting against the tampering with, and bypassing of, secured applications. The benefits derived from MAC would never be possible with the existing DAC operating systems. Many efforts have been devoted in defining and developing security model of trusted computer systems, requirements and architecture of secure operating systems. The results of some earlier research projects, such as Flask [3], and DTOS [4] were widely available in public. The emerging of more secure operating systems as commercial products and public domain software, e.g., HP-LX [5], SE-Linux [6], and Trusted Solaris, in recent years may indicate a new trend that attentions to the overall security of applications are duly focusing more on the root causes of the security of underneath operating systems. The remainder of this article begins with a general examination of potential risks resulting from the compromise of an application due to the lack of proper operating system security; and followed by a summary of the security model of DOD’s trusted computer system evaluation criteria. Then, based on the discussion of security requirements and general architecture of secure operatingsystems, a case study of the publicly available security enhanced Linux, SELinux, is presented at the end.

Security of Operating Systems. Most modern information computer systems provide concurrent execution of multiple applications in a single physical computing hardware (which may contain multiple processing units). Within such a multitasking, time-sharing environment, individual application jobs share the same resources of the system, e.g., CPU, memory, disk, and I/O devices, under the control of the operating system. In order to protect the execution of individual application jobs from possible interference and attack of other jobs, most contemporary operating systems implement some abstract property of containment, such as process (or task) and TCB (Task Control Block), virtual memory space, file, port, and IPC (Inter Process Communication), etc. An application is controlled that only given resources (e.g., file, process, I/O, IPC) it can access, and given operations (e.g., execution or read-only) it can perform. However, the limited containment supported by most commercial operating systems (MS Windows, various flavors of Unix, etc) bases access decisions only on user identity and ownership without considering additional security-relevant criteria such as the operation and trustworthiness of programs, the role of the user, and the sensitivity or integrity of the data. As long as users or applications have complete discretion over objects, it will not be possible to control data flows or enforce a system-wide security policy. Because of such weakness of current operating systems, it is rather easy to breach the security of an entire system once an application has been compromised, e.g., by a buffer overflow attack. Some examples of potential exploits from a compromised application are [5]:Use of unprotected system resources illegitimately. For example, a worm program launches attack via emails to all targets in the address book of a user after it gets control in a user account. Subversion of application enforced protection through the control of underneath system.

It is not possible to protect against malicious code of an application using existing mechanisms of most commercial operating systems because a program running under the name of a user receives all of the privileges associated with that user. Moreover, the access controls supported by the operating systems are so coarse — only two categories of users: either completely trusted super users (root) or completely un-trusted ordinary users. As the result, most system services and privileged applications in such systems have to run under root privileges that far exceed what they really needed. A compromise in any of these programs would be exploited to obtain complete system control. Model of Security Generally, in an access control based security model, there are set of objects, and set of subjects (a subject itself can also be an object). Every object has an associated security attribute, or security label; every subject also has a security label, or security clearance; and a defined set of control rule, or security policy that dictates which subject is authorized to access which object. For example, in military security model [7], a security label consists of two components: a security level with one of the four ratings: unclassified, confidential, secret, and top secret, where unclassified < confidential < secret < top secret, and “<” means “less sensitive than”; a set of zero or more categories (also known as compartments) that describe kinds of information, for instance, the names CRYPTO, NUCLEAR might mean information about cryptographic algorithms, and nuclear related technology. Given two security labels, (X, S1) and (Y, S2), (X, S1) is defined as being “at least as sensitive as” (Y, S2) iff X • Y and S2 Í S1. For example, (TOP SECRET, {CRYPTO, NUCLEAR}) > (SECRET, {CRYPTO}) where “>” means “more sensitive than”. In general, security labels are partially ordered. That is, it is possible for two labels to be incomparable, in the sense that neither is more sensitive than the other. For example, neither of the following is comparable to each other: (TOP SECRET, {CRYPTO}) (SECRET, {NUCLEAR}) A more generalized hierarchy of security classes (or levels) with a mathematical basis was presented by Bell and La Padula in 1973 [8]. In its effort to address computer security safeguards that would protect classified information in remote-access, resource-sharing computer systems, the National Computer Security Center (NCSC), later DOD (Department of Defense), published an official standard called “Trusted Computer System Evaluation Criteria” [1], universally known as “the Orange Book”. The Orange Book defines fundamental security requirements for computer systems and specifies a series of criteria for various levels of security ratings of a computer system based on its system design and security feature. A brief summary of all the ratings and their main characteristics are given as follows with a basic condition that each subsequent higher ratings must meet all the requirements of its lower ones.

D — Minimal Protection: no security is required; the system did not qualify for any of the higher ratings.

C1 — Discretionary Security Protection: the system must identify different users (or jobs) running inside the system, and provide mechanisms for user authentication and authorization to prevent unprivileged user programs from interfere each other (e.g., overwriting critical portions of the memory).

C2— Controlled Access Protection: the system meets additional security requirements than that of C1 that include access control at a per user granularity (access control for any subset of the user community); clearing of newly allocated disk space and memory; and ability of auditing (logging) for securityrelevant events such as authentication and object access, etc.

B1— Labeled Security Protection: the system must implement the Mandatory Access Control in which every subject and object of the system must maintain a security label, and every access to system resource (objects) by a subject must check for security labels and follow some defined rules.

B2— Structured Protection: few new security features are added beyond B1; rather the focus is on the structure (design) of the system to maintain greater levels of assurance so that the system behaves predictably and correctly (such as, a minimal security kernel, trusted path to user, and identified covert channels, etc).

B3— Security Domains: more requirements to maintain greater assurance that the system will be small enough to be subjected to analysis and tests, and not to have bugs that might allow something to circumvent mandatory access controls, e.g., support of active audit, and secure crashing, etc.

A1— Verified Design: no additional features in an A1 system over a B3 system; rather there are formal procedures for the analysis of the design of the system and more rigorous controls on its implementation. Most existing commercial operating systems are with the ratings of C2 or below.

Requirements of Secure Operation Systems. As discussed in above, most current operating systems provide discretionary access control, that is, someone who owns a resource can make a decision as to who is allowed to use (access) the resource. Moreover, because the lack of built-in mechanisms for the enforcement of security policies in such systems, the access control is normally a one-shot approach: either all or none privileges are granted, rarely supporting the “principle of least privilege” (without limiting the privileges a program can inherit based on the trustworthiness).

The basic philosophy of discretionary controls assumes that the users and the programs they run are the good guys, and it is up to the operating system to trust them and protect each user from outsiders and other users. Such perception could be extremely difficult to hold true and no longer be considered as secure enough for computer systems of “information era” with broad connectivity through the Internet and heavily commercialization of e-commerce services. Systems with stronger security and protection will require evolving from the approach of discretionary control towards the concept of mandatory (non-discretionary) control where information is confined within a “security perimeter” with strict rules enforced by the system about who is allowed access to certain resources, and not allow any information to move from a more secure environment to a less secure environment. Some of basic criteria or requirements of a secure operating system are discussed below.

Mandatory security — a built-in mechanism or logic within the operating system (often called system security module or system security administrator) that implements and tightly controls the definition and assignment of security attributes and their actions (security policies) for every operation or function provided by the system. Generally, a mandatory security will require:A policy independent security labeling and decision making logics. The operating system implements the mechanism, whereas the users or applications are able to define security policies.Enforcement of access control for all operations. All system operations must have permission checks based on security labeling of the source and target objects. Such enforcement requires controlling the propagation of access rights, enforcing fine-grained access rights and supporting the revocation of previously granted access rights, etc. The main security controls include permission or access authorization, authentication usage, cryptographic usage, and subsystem specific usage, etc.

Conclusions. With ever growing security alerts and CERT Advisory for systems like Microsoft Windows and the ordinary Linux, people must be wandering how such games of cat-mouse-catching would ever be ended, and if there could be any better way to address the root causes of many of general vulnerabilities of information systems. The approach covered in this article — executing applications from a strongly guarded, secure operating system — certainly opens an alternative frontier in battling with many of existing cyber-space threats of the real world. Although, the approach of using secure operating systems will not be a panacea for all the dangers of current cyber space, and the security of individual applications may still suffer from the vulnerabilities of their own, with the strong containment of a secure operation system, the damages caused from a compromise within one application would be much localized, and the impacts among various applications could be much well controlled.

As a demonstration of how mandatory access control can be integrated into a popular, main-stream operating system, the release of SE-Linux to general public assures that the usage of secure operating systems is not necessarily an expensive endeavor limited only to academic and defense related institutions, and encourages further efforts in research and development of secure operating systems. Not much testing result has been reported regarding the performance impacts and effectiveness of MAC of SE-Linux. It would be interesting to see some experimental deployment and test results using SE-Linux with real-world applications, such as Web servers for e-commerce services.

References:

1. DOD 5200.28-STD, “DOD Trusted Computer System Evaluation Criteria” (Orange Book), 26 December 1985, http://www.radium.ncsc.mil/tpep/library/rainbow/5200.28-STD.pdf.
2. P. A. Loscocco, S. D. Smalley, P. A. Muckelbauer, R. C. Taylor, S. J. Turner, and J. F. Farrell, “The Inevitability of Failure: The Flawed Assumption of Security in Modern Computing Environments”, Proceedings of the 21st National Information Systems Security Conference, pages 303–314, Oct. 1998. http://www.nsa.gov/selinux/doc/inevitability.pdf.
3. Flask: Flux Advanced security Kernel, http://www.cs.utah.edu/flux/fluke/html/flask.html.
4. DTOS Technical Reports, http://www.securecomputing.com/randt/HTML/technical-docs.html.
5. Chris Dalton and Tse Huong Choo, “An Operating System Approach to Securing E-Services”, Communications of the ACM, V. 44, No. 2, p. 58, 2001.
6. Security Enhanced Linux, http://www.nsa.gov/selinux/index.html.
7. Charlie Kaufman, Radia Perlman, and Mike Speciner, “Network Security: Private Communication in a Public World”, PTR Prentice Hall, Englewood Cliffs, New Jersey, 1995.
8. D. E. Bell and L. J. La Padula, “Secure Computer Systems: Mathematical Foundations and Model”, Technical Report M74–244, The MITRE Corporation, Bedford, MA, May 1973.