Episode 233

Posted on Friday, Aug 2, 2024
This week we take a look at the recent Crowdstrike outage and what we can learn from it compared to the testing and release process for security updates in Ubuntu, plus we cover details of vulnerabilities in poppler, phpCAS, EDK II, Python, OpenJDK and one package with over 300 CVE fixes in a single update.

Show Notes

Overview

This week we take a look at the recent Crowdstrike outage and what we can learn from it compared to the testing and release process for security updates in Ubuntu, plus we cover details of vulnerabilities in poppler, phpCAS, EDK II, Python, OpenJDK and one package with over 300 CVE fixes in a single update.

This week in Ubuntu Security Updates

462 unique CVEs addressed

[USN-6915-1] poppler vulnerability (01:35)

  • 1 CVEs addressed in Jammy (22.04 LTS), Noble (24.04 LTS)
  • Installed by default in Ubuntu due to use by cups
  • PDF document format describes a Catalog which has a tree of destinations - essentially hyperlinks within the document. These can be either a page number etc or a named location within the document. If open a crafted document with a missing name property for a destination - name would then be NULL and would trigger a NULL ptr deref -> crash -> DoS

[USN-6913-1] phpCAS vulnerability (02:26)

  • 1 CVEs addressed in Focal (20.04 LTS), Jammy (22.04 LTS)
  • Authentication library for PHP to allow PHP applications to authenticates users against a Central Authentication Server (ie. SSO).
  • When used for SSO, a client who is trying to use a web application gets directed to the CAS. The CAS then authenticates the user and returns a service ticket - the client then needs to validate this ticket with the CAS since it could have possibly been injected via the application. To do this, pass the ticket along with its own service identifier to CAS - and if this succeeds is provided with the details of which user was authenticated etc.
  • For clients, previously would use HTTP headers to determine where the CAS server was to authenticate the ticket. Since these can be manipulated by a malicious application, could essentially redirect the client to send the ticket to the attacker who could then use that to impersonate the client and login as the user.
  • Fix requires a refactor to include an additional API parameter which specifies either a fixed CAS server for the client to use, or a mechanism to auto-discover this in a secure way - either way, applications using phpCAS now need to be updated.

[USN-6914-1] OCS Inventory vulnerability

  • 1 CVEs addressed in Jammy (22.04 LTS)
  • Same as above since has an embedded copy of phpCAS

[USN-6916-1] Lua vulnerabilities (04:44)

  • 2 CVEs addressed in Jammy (22.04 LTS)
  • Heap buffer over-read and a possible heap buffer over-flow via recursive error handling - looks like both require to be interpreting malicious code

[USN-6920-1] EDK II vulnerabilities (05:04)

[USN-6928-1] Python vulnerabilities (05:49)

  • 2 CVEs addressed in Focal (20.04 LTS), Jammy (22.04 LTS)
  • Memory race in the ssl module - can call into various functions to get certificate information at the same time as certs are loaded if happening to be doing a TLS handshake with a certificate directory configured - all via different threads. Python would then possibly return inconsistent results leading to various issues
  • Occurs since ssl module is implemented in C to interface with openssl and did not properly lock access to the certificate store

[USN-6929-1, USN-6930-1] OpenJDK 8 and OpenJDK 11 vulnerabilities (06:52)

[USN-6931-1, USN-6932-1] OpenJDK 17 and OpenJDK 21 vulnerabilities (07:11)

[USN-6934-1] MySQL vulnerabilities (07:29)

[USN-6917-1] Linux kernel vulnerabilities (07:57)

[USN-6918-1] Linux kernel vulnerabilities

[USN-6919-1] Linux kernel vulnerabilities

[USN-6922-1] Linux kernel vulnerabilities

[USN-6923-1, USN-6923-2] Linux kernel vulnerabilities

[USN-6921-1, USN-6921-2] Linux kernel vulnerabilities

[USN-6924-1, USN-6924-2] Linux kernel vulnerabilities

[USN-6925-1] Linux kernel vulnerability

  • 1 CVEs addressed in Trusty ESM (14.04 ESM)
  • 3.13 - generic, lowlatency, server, virtual

[USN-6926-1] Linux kernel vulnerabilities

[USN-6927-1] Linux kernel vulnerabilities

Goings on in Ubuntu Security Community

Discussion of testing for security updates in light of CrowdStrike (11:20)

  • Recent outage of over 8 million Windows machines running CrowdStrike Falcon
  • Initially very little information on what happened - CS have now released more details about the apparent testing that was done but clearly were never actually testing the combination of Windows + Falcon + Rapid Response Content otherwise would have observed this failure immediately
  • Also clearly didn’t have any kind of staged/phased update process in place either
    • If you want to read a good analysis of the response from CS, https://verse.systems/blog/post/2024-07-25-parsing-crowdstrikes-post/
      • Toby Murray (full disclosure, my brother) - Associate Professor and Co-Lead of Computer Science Research Group at School of Computing and Information Systems, University of Melbourne, Director, Defence Science Institute (Vic & Tas)
    • Future plans from CS now include gradual deployment of rules with “canaries” etc and then increased testing:
      • Local dev testing, content update testing, stress, fuzz, fault-injection, stability and interface testing
      • Toby (not surprisingly as an expert in formal software verification) advocates for a formal approach to validating rules and in-kernel code etc
  • What can we learn from this for Ubuntu?
    • Formal methods might be tractable for a large company like CS who is developing a single, specific product like Falcon (particularly if they can reduce the size of their kernel module), this is not the case for a Linux distribution like Ubuntu which collates over 30,000 different open source software projects
      • over 4TB of source code across the various releases
  • Instead have to take the pragmatic approach of thorough testing
    • For regular SRUs - detailed review by SRU team including a thorough test plan, cross-package testing via Autopkgtest plus a minimum 7 day “soak” testing in the proposed pocket of the release before being pushed into the -updates pocket
    • Once in -updates, Phased Updates implements the gradual deployment model - you can check the progress of various updates at https://ubuntu-archive-team.ubuntu.com/phased-updates.html
      • Watches for increased error reports via errors.ubuntu.com (captured via apport/whoopsie) and if detected stops the release of the package to users
  • Compare that to the process for Security updates
    • Separate -security pocket in the archive which packages get published to immediately
      • No standardised review by separate team
        • instead adhoc reviews within the security team
      • No documented test plan per update
        • instead thorough test procedures including:
          • checking for any changes in the build log (e.g. new compiler warnings/errors) and comparing the difference between the generated binaries (e.g. new / changed / missing symbols - ABI breaks)
          • testing of the patched code including stepping through it with a debugger
          • running any existing PoC or creating one if none exists and is feasible
          • running any existing unit/integration tests within the package (including dep8/autopkgtests)
          • test apt upgrade of the package is smooth
          • QA regression testing scripts - maintained by the security team, implement various regression tests and system-level tests for different packages to exercise them in various different configurations
      • Cross-package testing via security-britney - instance of the autopkgtest infrastructure that runs against the public Ubuntu Security Proposed PPA (and we have a similar internal instance for the different private PPAs we use for embargoed updates or ESM etc)
      • No phased updates - instead immediate updates via specific security.ubuntu.com archive, combined with unattended-upgrades
        • designed to deliver security updates as soon as possible to remediate issues
  • In general, I would argue that the process we have in place results in more thorough testing for security updates - particularly checking for anything anomalous like new compiler warnings / symbols / unexpected changes in binaries etc as well as more thorough, standardised testing for packages through the QA Regression Testing repo scripts
  • However, the lack of phased/progressive updates combined with the separate security.ubuntu.com archive and unattended-upgrades on by default, means any security update is delivered to Ubuntu users within 24 hours (on average) - BUT then any regression is also rolled out to all users in 24 hours as well
  • As such, kicking off discussions around possible changes to our deployment strategy to potentially introduce some more guard rails on the deployment side
  • If you have any thoughts, please let us know

Get in contact