IT Systems Engineering - Unix Engineering Service Outline

Customer & IT Systems Engineering responsibility matrix

Customer-owned equipment, and the software run on said equipment, remains the property and responsibility of the customer. It is the customer's responsibility to ensure that their equipment and software is in compliance with University and campus IT policies and standards, including but not limited to the campus' Acceptable Use Standard and Secure Computing Standard for Servers. In particular, it is the customer's responsibility to ensure that their equipment is being used for appropriate and allowable purposes and that their equipment is running current, supported operating systems and software. It is also the customer's responsibility to ensure that operating system and application security updates are applied in compliance with the Vulnerability Management Standard. Customers are responsible for the data they and their constituents put on the server, including web content, user-installed software, data management and auditing, and data classification reporting

VM Base Pricing

The base price for a VM includes the following:

• Regular, automatic patching
• Tri-Annual reboots
• Software Installation
• Strict supported software stack list for each OS version

Extra Charges

• Manual patching and rebooting of systems can be done every three months at $86/month price increase. 2 hours per instance, 4 times a year, is $1040/year.
  • If this is required by a system owner, they must subscribe to security patches and those will still be applied automatically.
  • Can automatically apply security patches in certain windows, so instead of automatic whenever they come out, something like every second Saturday.
• System must be able to withstand security scanning.
• 24x7 support (instead of 11x5) is $20/month

Billing and Payment

Billable items – including both recurring charges and one-time consulting charges – can only be paid via speedtype on a monthly schedule. IT Systems Engineering - Unix Engineering (SEUE) does not offer other billing schedules, like quarterly or yearly. New system administration charges are effective at the beginning of the month (i.e. the 1st of the month) closest to when the system goes live.

Visit the Systems Engineering Pricing page for the current list of costs for this service. 

Maintaining System Tracking Information

System owners are responsible for reviewing and updating system tracking information for managed systems under their name on a yearly basis. This includes the owner’s email address, email addresses of authorized technical contacts, reported system data classification, emergency contact information (for 24x7 support, only), and system description. 
  
System tracking information will be emailed to the system owner at the beginning of each fiscal year (July 1st). System owners are responsible for confirming the current status of their system tracking information or submitting changes by the end of the month.

OS Release Cycle

Beginning with RHEL 8, SEUE manages RedHat Enterprise Linux (RHEL) distributions which are currently under “Full Support” from RedHat, including RHEL Full Support and RHEL Maintenance Support (see https://access.redhat.com/support/policy/updates/errata).

System owners with RHEL 7 systems are encouraged to upgrade immediately as portions of the system may not be receiving security updates now that RHEL 7 is in the Maintenance Support phase. Only select components are maintained by RedHat during the Maintenance Support phase.

Other Unix or Linux distributions, including RHEL-like distributions, are not supported by SEUE. This includes but is not limited to Debian, Suse, Rocks, Ubuntu, and Solaris.

Customers that need to continue running systems past the RHEL Full Support phase, known as Maintenance Support, may do so for until the end of the Maintenance Support phase if they meet the following criteria.

• System owners must be pre-approved by OIS one year prior to the end of the RHEL Full Support phase for their RHEL OS version.
• All software components installed on the system must be receiving security patches during the planned remaining lifetime of the system.
• System owners must complete the formal risk acceptance process at least one year prior to the end of the Full Support phase for their current RHEL release.
• System owners must submit a formal upgrade plan at least one year prior to the end of the Full Support Phase for their current RHEL release.

Current RHEL versions and support dates

RHEL systems will not be supported into the Extended Lifecycle Support phase under any circumstances.

System Upgrades

Periodic, planned system upgrades are necessary to keep systems running on supported OS versions. System upgrades require the involvement of SEUE, system owners, and technical contacts. Furthermore, additional parties may be required to participate in system upgrade work in some circumstances.

Due to the number of systems supported by SEUE, system upgrades must be scheduled in advance. Work will not begin until all required parties have verified their availability during the scheduled upgrade window.

System owners can request a system upgrade at any time to a currently supported RHEL version, even if it is before the end of the RHEL Full Support date.

Hardware Requirements (for non VM systems)

Hardware systems under management of SEUE must meet the following requirements:

• Must be a Dell brand, rack-mountable, server with an iDRAC enterprise controller card.
• Must reside in an OIT supported data center – either COMP, SPSC, or INFO data centers.
• Must be under active warranty support at all times from an approved warranty provider.
• Approved warranty providers are Dell and ServiceExpress.
• Must be less than 11 years of age

System owners are responsible for paying warranty costs for their hardware system(s). SEUE will notify the system owners and technical contacts when a warranty is close to expiration and will supply a warranty quote to the system owner. SEUE will not pay for warranty costs on behalf of the system owner nor offer any repayment options for renewing warranty agreements on their behalf.

Additionally, management of hardware systems are charged differently than management of VMs. See the Billing and Payment section for more information.

Security Compliance

All IT resources owned and operated by the university must adhere to certain policy requirements. Additionally, IT resource users must adhere to specific responsibilities. Failure to comply with any CU-System wide, CU Boulder Campus, or CU OIT policies may result in access restriction to a server and/or service. Extreme cases may also result in complete termination of a server and/or service.

Policy and Standards Compliance

Know your responsibilities as they pertain to the following policies and standards

• IT Security Program: Serves as the core for the university's information security activities and provides general guidance
• Data Classification: Classifying or labeling university information helps determine minimum security requirements necessary to keep it safe
• Data Governance: Ensures university data is managed as a material asset
• Systemwide Security Baseline Standards: Provides guidelines for selecting and specifying security controls for organizations and information systems
• Systemwide High Impact Security Standards: Security standards and requirements for protecting highly confidential information when processed, stored, or transmitted.
• Website & Web Application Security: Technical and procedural standards for development of web sites and web applications for CU Boulder entities.
• Secure Computing Standard for Servers: Minimum requirements for all University servers to ensure the integrity and security of University Data and the shared information technology environment, including networks, services, and systems.

Security Patching 

In addition to the policies and standards above, all SEUE managed systems will receive automatic patches. Two different security patching and reboot schemes are available to choose from:

1. Automatic patches with tri-annual automatic reboots
   • Automatic patches – consisting of all available patches, including security patches, bug fixes, and feature enhancements – are applied automatically.
   • Tri-annual automatic reboots that take place during the CU Boulder winter holiday break, spring break, and the week preceding the first day of the fall semester.
2. Automatic security-only patches with quarterly manual reboots
   • Security-only (i.e. only patches which fix a known security issue) are applied automatically.
   • All other patches – including bug fixes and feature enhancements – are applied manually during the quarterly reboot window.
   • Configuration management changes are applied manually during the quarterly reboot window.
   • Quarterly reboot windows can be one of the following: 
     i. January, April, July, October 
     ii. February, May, August, November 
     iii. March, June, September, December
   • Quarterly reboots may not occur during campus holidays.
   • Services which have a Dev/Test/Production (or similar) pipeline may space out system patching and reboots during the reboot window.
   • Automatic security-only patching with quarterly manual reboots incurs an additional cost of $80 per month per system.

For software installed by the system owner (or any past or present technical contacts) the system owner is responsible for ensuring timely updates are applied to these software components. This is critical as there are often security patches released on a regular basis. We recommend subscribing to the relevant updates to be sure you are notified of new releases.

Security Scans

Monthly on-campus and off-campus security scans will be run against all SEUE managed systems. Systems which are unable to handle the security scan load must purchase additional compute resources and/or be improved (at the system owner’s expense) to handle the additional load of security scans. Any vulnerabilities identified by the security scan must be remediated either by SEUE or the system owner (whichever holds responsibility for the affected component) per the vulnerability remediation expectations outlined below.

Vulnerability Remediation Expectations

Expected timeframes for completion of remediation will vary based on the data criticality and sensitivity of the system and the criticality of the vulnerability. The following expectations apply for low impact systems which are accessible from the Internet:

• Confirmed Urgent Vulnerability (level 5) with serious known exploit – 48 hours to remediate
• Confirmed Urgent Vulnerability (level 5) - 14 days to remediate
• Confirmed Critical Vulnerability (level 4) – 30 days to remediate

If remediation is not completed within the expected time or a risk acceptance decision (following the process outlined in section D of the Identification and Management of Security Flaws in IT Systems standard) is not completed, then it will be necessary to block access from the Internet until remediation is complete. If the IT Security Office does not receive a response to the initial notification of the vulnerability it may be necessary to block access from the Internet until remediation is complete.

The following expectations apply for high-risk systems (e.g., PCIDSS or systems maintaining highly confidential data):

• Confirmed Urgent Vulnerability (level 5) – Business must be suspended until remediated.
• Confirmed Critical Vulnerability (level 4) – Business must be suspended until remediated.
• Confirmed Serious Vulnerability (level 3) – Action plan submitted to campus ISO with remediation occurring within 180 days. If found on 3 plus consecutive plans, campus ISA and treasury will be notified for possible suspension.

If remediation is not completed within the specified time network access may be revoked. In the case of PCIDSS systems the campus ISA and University Treasury will determine if the merchant account is to be suspended.

The IT Security Office will confirm remediation with a verification scan one day after the expected timeframe for completion.

24x7 Support

For an additional cost of $20 per month systems may opt into 24x7 support. 24x7 support includes 24x7 system monitoring as well as 24x7 incident response. 24x7 support does not include off-hours non-incident response work. All off-hours non-incident work requested of SEUE, for example scheduling an upgrade or manual reboot outside of the hours of 8AM to 6PM (Boulder local time), will incur a charge of $230 per hour, billed in 30 minute increments.

Website Development and Maintenance

Website development and maintenance is the sole responsibility of the system owner. SEUE does not offer website development nor software development services. System upgrades may require software development or refactoring to accommodate new software versions or software replacements offered by the updated operating system. Development work must be factored into the upgrade timeline by the system owner.

System owners are responsible for keeping all website code and frameworks under their control up-to-date for all known security vulnerabilities per the Security Compliance section.

System owners are responsible for monitoring the content of their website, properly classifying this content, and carry a significant responsibility to protect data and prevent unauthorized use. System owners are responsible for reporting data classifications to SEUE and collaborating with SEUE to secure data on their system(s).

Handling of Requests and Incidents

SEUE separates incoming work into two different categories, incidents and requests. Incidents are defined as issues with pre-established software and/or services running on an existing SEUE managed system. Requests are all other types of work, including new software installations and configuration changes.

In order to ensure business service continuity and to handle incoming requests in a fair manner, work is queued as follows: incidents supersede requests, incidents are handled in order of impact and urgency, and requests are handled first-come-first-serve. Therefore, all requests should be submitted well in advance of any known deadlines.

Consulting Services

Consulting services can be requested from SEUE at an hourly rate of $130/hour for Linux system administration work on unmanaged systems, during the hours of 8AM-6PM (Boulder local time). Consulting requests will be handled in the same manner as all other work requests – on a first-come-first-serve basis. Consulting work is contingent on SEUE’s availability and consulting requests may be declined at any time.

Terms and Definitions

• Active warranty support: A current contract with a hardware technical support service provider (e.g. Dell or Service Express), that provides hardware troubleshooting assistance and replacement parts.
• Business hours: Monday through Friday from 8AM to 6PM (Boulder local time), except for campus holidays.
• COMP: The Computing Center building. This building houses OIT offices on East Campus. This building serves as the main data center for campus.
• Data classification: The type of data residing on a managed system. Can be one of; highly confidential information, confidential information, or public information. If multiple types of data reside on the managed system, only the most sensitive information classification needs to be reported. Additional information about each data type and their definitions can be found on the OIS website.
• Hardware system: A physical computer. Not a virtual machine.
• iDRAC enterprise controller card: The enterprise version of Dell’s Integrated Dell Remote Access Controller (iDRAC), which provides remote access for server hardware management, firmware updates, and hardware troubleshooting.
• INFO: Home of the Department of Information Science’s faculty and administrative offices and CMCI student space. Formerly the Technology Learning Space (TLC).
• Managed system: A server under the management of SEUE.
• Off-hours: Any time not falling within the scope of business hours as defined above.
• OIS: The Office of Information Security (OIS) is based out of System Administration and partners with the campuses to provide services and expertise to support confidentiality, integrity and availability for data universitywide. (source: https://www.cu.edu/security/about)
• PCIDSS: Payment Card Industry Data Security Standard. An information security standard for organizations that handle branded credit cards from the major card schemes.
• RedHat Enterprise Linux (RHEL): Red Hat Enterprise Linux is a commercial open-source Linux distribution developed by Red Hat for the commercial market. (source: https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux)
• RHEL Full Support: During the Full Support Phase, Red Hat defined Critical and Important Security errata advisories (RHSAs) and Urgent and Selected (at Red Hat discretion) High Priority Bug Fix errata advisories (RHBAs) may be released as they become available. Other errata advisories may be delivered as appropriate. (source: https://access.redhat.com/support/policy/updates/errata#Full_Support_Phase)
• RHEL Maintenance Support: During the Maintenance Support Phase for Red Hat Enterprise Linux Version 8 and Maintenance Support 2 Phase for Red Hat Enterprise Linux version 6, and 7, Red Hat defined Critical and Important impact Security Advisories (RHSAs) and selected (at Red Hat discretion) Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available. Other errata advisories may be delivered as appropriate. (source: https://access.redhat.com/support/policy/updates/errata#Maintenance_Support_2_Phase)
• RHEL Extended Lifecycle Support (ELS): Extended Life-cycle Support (ELS) is an optional Add-On subscription for certain Red Hat Enterprise Linux subscriptions. The ELS Add-on is available during the Extended Life Phase for Red Hat Enterprise Linux. (source: https://access.redhat.com/support/policy/updates/errata#Extended_Life_Cycle_Support)
• Risk acceptance process: The risk acceptance [process] is to be used in instances where the institutional risk is likely to exist for more than three (3) months and a risk analysis has been performed, identifying the potential impact of the risk as high to the University. The potential impact is high if the loss of confidentiality, integrity, or availability could be expected to have a severe or catastrophic adverse effect on organizational operations, organizational assets, or individuals. (source: https://www.cu.edu/security/risk-acceptance-process)
• SEUE: IT Systems Engineering - Unix Engineering. Department within the Office of Information Technology which manages RedHat Enterprise Linux servers for various departments across campus.
• SPSC: Space Science Building. This is one of two LASP locations in the CU East Campus Research Park. The SPCS building is host to LASP's Science Division, Office of Communications & Outreach, and a campus data center.
• System owner: The person responsible for overseeing the end-user services run on the managed system.
• Technical contact(s): The person(s) who help maintain the end-user services running on the managed system. Technical contacts are authorized to perform technical tasks and make technical decisions on the system owner’s behalf.
• Virtual Machine (VM): The virtualization/emulation of a computer system. Virtual machines are based on computer architectures and provide functionality of a physical computer. (source: https://en.wikipedia.org/wiki/Virtual_machine)