Watching xz unfold from afar

Early Friday morning (3/29/2024) I am checking my daily dose of Hacker News and the top article is rising fast titled - "Backdoor in upstream xz/liblzma leading to SSH server compromise" and reading the article tells a scary tale that breaks down the following:

• Andres Freund is investigating some Postgres performance issues and notices SSH using more CPU than normal.
• He discovers injected code coming from a dependency known as xz / liblzma.
• He further discovers the requirements it needs to execute and how it interacts with the RSA portion of authentication during ssh.
• This sets off many alarms and he reaches out to the many different organizations & people to validate his findings.

They all appear to be true and the last 48 hours have been hectic. So at first I wanted to figure out what xz even was - as I only vaguely remembered it being some compression library.

So it was a bit more than I thought, but then I was curious how this got related to ssh, because I didn't see the package listed as a dependency of openssh. I pulled down the openssh Debian package and grepped for "xz".

debian/changelog:  * Remove dh_builddeb override to use xz compression; this has been the
debian/changelog:  * Fix dh_builddeb invocation so that we really use xz compression for
debian/changelog:  * Use xz compression for binary packages.

The only few results did not look helpful as just because you use xz compression does not directly mean invocation of the software at runtime. So Debian must have added another dependency onto ssh to bring in this package. Thankfully due to the patch approach Debian makes on top of upstream software - its incredibly easy to find the changes.

Knowing we are adding a dependency it probably has to be adjusting the build configuration files. So another grep scoped to configure.ac and we have 4 patches to check:

• gssapi.patch - Adds GSSAPI key exchange support.
• restore-tcp-wrappers.patch - Restore TCP wrappers support.
• systemd-readiness.patch - Add systemd readiness notification support.
• systemd-socket-activation.patch - Support systemd socket activation.

Going down the list we don't see anything until the systemd patch which appears to add support for systemd.

Peeking around systemd it didn't take long - their build system has support for both xz and liblzma. So now I understood how the ssh process was being hijacked during the security disclosure. We have in this specific example openssh patched with systemd support that depends on lzma.

Now that the chain of dependencies is understood - I wanted to review how the malware was injected and who did it.

The researcher (Andres) called out the following commit as the introduction of the exploit, which looks innocent as it adds new test files. However any engineer peeking that commit could tell that binary files added with no real code reference to them is odd.

This is where the story gets suspicious - once the malware was introduced, it didn't really work. This led to the malware author fixing issues quickly with commits such as:

liblzma: Fix false Valgrind error report with GCC.

With GCC and a certain combination of flags, Valgrind will falsely
trigger an invalid write. This appears to be due to the omission of
instructions to properly save, set up, and restore the frame pointer.

The IFUNC resolver is a leaf function since it only calls a function
that is inlined. So sometimes GCC omits the frame pointer instructions
in the resolver unless this optimization is explictly disabled.

This fixes https://bugzilla.redhat.com/show_bug.cgi?id=2267598.

liblzma: Use attribute no_profile_instrument_function with ifunc.

Thanks to Sam James for determining this was the attribute needed to
workaround the GCC bug and for his version of the patch in Gentoo.

Build: Require attribute no_profile_instrument_function for ifunc usage.

Using __attribute__((__no_profile_instrument_function__)) on the ifunc
resolver works around a bug in GCC -fprofile-generate:
it adds profiling code even to ifunc resolvers which can make
the ifunc resolver crash at program startup. This attribute
was not introduced until GCC 7 and Clang 13, so ifunc won't
be used with prior versions of these compilers.

This bug was brought to our attention by:

    https://bugs.gentoo.org/925415

And was reported to upstream GCC by:

    https://gcc.gnu.org/bugzilla/show_bug.cgi?id=11411

Without the idea of a malware in your eye could these issues and fixes be okay? I'm not well enough versed to make that determination, but llvm comments are torn on this subject as they discuss in an issue thread opened by our alleged malicious actor.

tukaani-project/xz@82ecc53 is a legitimate change that exposes two problems:
// snipped
We should be careful and don't consider all xz commits malicious.

On the flip side another user takes a different stance.

I disagree, we need to be extra careful with changes associated with the xz project right now. There's no reason why we need to get this change committed quickly. It should get extra scrutiny. [source]

Once the above commits are in - Jia Tan issues another update to the malware to get it working. They hide their tracks with once again a misleading commit message.

Tests: Update two test files.

The original files were generated with random local to my machine.
To better reproduce these files in the future, a constant seed was used
to recreate these files.

Along that same path Google offers an open platform called OSS-Fuzz, which continually fuzzes open software to detect errors that could have security implications. As of today its detected nearly 10,000+ vulnerabilities and fixed over 36,000 bugs.

This tool does its fuzzing and informs the maintainers of the software to provide them with output of failures detected. The maintainers can also put forward fixes as shown below to ensure the tool operates against their project. So something looked a bit odd to me when I noticed a build failure with xz in which our alleged malware author resolved it – by disabling something.

Once again I'm not well enough versed in this field to determine if this is the right fix or not, but it seems extremely odd disabling something on an alleged incompatibility when the other recent comments says there is none. Especially when that failure was caused by the introduction of code (IFUNC) that would then be leveraged in the malicious code.

However, the timing on this was almost a year ago so now I was nervous how this author (JiaT75) was a primary contact for the xz project. Was their account hijacked or has this individual been working on something nefarious for years.

I dug further and found the pull request where this new author claimed himself to be the new primary maintainer with the previous maintainer (Larhzu) approving it. So now I wanted to figure out how this transition occurred.

The mailing lists for this seem extremely odd when you look back at it. I'll summarize a thread roughly that started in May of 2022.

Dear XZ Java Community

Is XZ for Java still maintained? I asked a question here a week ago
and have not heard back [source]

To which the maintainer (Larhzu) responds with:

I saw. I have lots of unanswered emails at the moment and obviously
that isn't a good thing. After the latest XZ for Java release I've
tried focus on XZ Utils (and ignored XZ for Java), although obviously
that hasn't worked so well either even if some progress has happened
with XZ Utils. [source]

Some new guy (Jigar Kumar) jumps into the chain with toxicity.

Progress will not happen until there is new maintainer. XZ for C has sparse
commit log too. Dennis you are better off waiting until new maintainer happens or fork yourself. Submitting patches here has no purpose these days. The current maintainer lost interest or doesn't care to maintain anymore. It is sad to see for a repo like this. [source]

The maintainer responds again with:

I haven't lost interest but my ability to care has been fairly limited
mostly due to longterm mental health issues but also due to some other
things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and
perhaps he will have a bigger role in the future, we'll see.

It's also good to keep in mind that this is an unpaid hobby project. [source]

The new guy (Kumar) responds hastily again.

With your current rate, I very doubt to see 5.4.0 release this year. The only
progress since april has been small changes to test code. You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo.
Why wait until 5.4.0 to change maintainer? Why delay what your repo needs? [source]

Finally the maintainer responds with:

As I have hinted in earlier emails, Jia Tan may have a bigger role in
the project in the future. He has been helping a lot off-list and is
practically a co-maintainer already. :-) I know that not much has
happened in the git repository yet but things happen in small steps. In
any case some change in maintainership is already in progress at least
for XZ Utils. [source]

We have the beginning proof that Jia Tan has been helping out a lot with this project working up to a maintainer role. The paranoia in me is racing now - did this individual understand that xz was a dependency of systemd and thus in certain circumstances of ssh?

With a quick search of Jigar Kumar I only see 6 messages with them all being between April 2022 and June 2022. It seems almost like a sock-puppet account that solely added a rushed attitude and perception to the chain to push the original maintainer to add another. It seems a campaign was occurring to really convince Lasse to share some responsibilities on this project.

So now that I understood how this individual became a maintainer I wanted to figure out how it got into operating system distributions. I stumbled upon a Debian thread where it was in discussion for the 5.6.1 xz release to be included. What is odd about this is the requestor is not the maintainer (Jia Tan).

Hans Jansen <hansjansen162@outlook.com>
Dear mentors,

I am looking for a sponsor for my package "xz-utils":

// snipped

This new version fixes a valgrind bug with liblzma that outputs a false
warning that could affect existing testing frameworks for packages that
test with valgrind requiring a specific output. This release only fixes
bugs.

Another user makes an odd comment in support of this upgrade.

krygorin4545 <krygorin4545@proton.me>
Also seeing this bug. Extra valgrind output causes some failed tests for me. Looks like the new version will resolve it. Would like this new version so I can continue work.

Add another...

misoeater91@tutamail.com
I noticed this last week and almost made a valgrind bug. Glad to see it being fixed.

Thanks Hans!

A user with a Debian email is right to be suspicious and calls it out.

Thorsten Glaser <tg@debian.org>
Very much *not* a fan of NMUs doing large changes such as
new upstream versions.

But this does give us the question, what’s up with the
maintenance of xz-utils? Same as with the lack of security
uploads of git, which you also maintain, are you active?
Are you well?

For those unaware the NMU call out is a "Non Maintainer Upload" which helps dictate a process (versioning, criteria, etc) when it occurs. Of course anyone can present patches, but a non-maintainer uploading a new upstream version can be perceived as perfectly normal (in the case of abandonment) or odd (in this case of this). Either way it generally means a greater care into researching the change/upgrade.

Ultimately the 5.6.1 release was merged into unstable, which presumably fixed the malware we previously examined. Thankfully the "fixed" malware never made it to testing - only 5.6.0.

Source: xz-utils
Source-Version: 5.6.1-1
Done: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>

That was of course quickly reverted a day later all the way back to the 5.4.5 version and rushed into testing.

What was odd again in this situation much like we saw with Jigar Kumar is we have three users that came out of nowhere to draw support for this inclusion of malware or drive it.

• hjansen (author)
• krygorin4545 (comment)
• misoeater91 (comment)

Some of these accounts were made within days of the message while the others made in the last year. They seem and appear to be involved with this malware scheme acting like additional sock puppets.

Maybe this is more than one person behind the scenes splitting up to hit all the distributions as you can see Jia Tan themselves attempting to get the updated (malicious) version into Ubuntu as well.

So now I was curious when Jia Tan was created on GitHub and scrolled all the way back.

This user wasn't a very old account being only created in January of 2021 and one of their first public merged pull requests is also under investigation. It fixed a bug while introducing a possible more severe one in not sanitizing terminal escape sequences. The community worked together and re-fixed the potentially vulnerable patch by Jia Tan. Was this going to be another vector? The project (libarchive) does have another hit on usage in systemd.

I scroll the user's history on GitHub and I'm blown away - commits and fixes to projects all over - some seemingly great and normal while others being reverted as quick as possible.

Take another commit by the malicious actor - it seemingly looks alright on a quick peek.

From: Jia Tan <jiat0218@gmail.com>
Date: Mon, 26 Feb 2024 15:02:06 +0000 (+0800)
Subject: Build: Fix Linux Landlock feature test in Autotools and CMake builds.
X-Git-Url: https://git.tukaani.org/?p=xz.git;a=commitdiff_plain;h=328c52da8a2bbb81307644efdb58db2c422d9ba7

Build: Fix Linux Landlock feature test in Autotools and CMake builds.

The previous Linux Landlock feature test assumed that having the
linux/landlock.h header file was enough. The new feature tests also
requires that prctl() and the required Landlock system calls are
supported.
---

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 76700591..d2b1af7a 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -901,10 +901,29 @@ endif()
 
 # Sandboxing: Landlock
 if(NOT SANDBOX_FOUND AND ENABLE_SANDBOX MATCHES "^ON$|^landlock$")
-    check_include_file(linux/landlock.h HAVE_LINUX_LANDLOCK_H)
+    # A compile check is done here because some systems have
+    # linux/landlock.h, but do not have the syscalls defined
+    # in order to actually use Linux Landlock.
+    check_c_source_compiles("
+        #include <linux/landlock.h>
+        #include <sys/syscall.h>
+        #include <sys/prctl.h>
+.
+        void my_sandbox(void)
+        {
+            (void)prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
+            (void)SYS_landlock_create_ruleset;
+            (void)SYS_landlock_restrict_self;
+            (void)LANDLOCK_CREATE_RULESET_VERSION;
+            return;
+        }
+
+        int main(void) { return 0; }
+        "
+    HAVE_LINUX_LANDLOCK)

Maybe you skimmed it and missed the mistake. A little innocuous dot (.) hiding in the check_c_source_compiles function. This makes the source invalid so of course it fails compilation. This leads to a situation where the landlock sandbox will always be disabled as the source will never compile cleanly.

It seems in under a day multiple malicious commits have been discovered. This user and the crowd of sock puppets around them are under major investigation by the masses. The story is unfolding at a massive pace - for everything this user did for the past ~2.5 years now seems risky.

From registering on GitHub to joining IRC (Libera) chat - everything started just a few years ago and none of it can be trusted. Chats are being scanned, mailing lists searched and commits being reviewed.

NickServ: Information on jiatan (account jiatan):
NickServ: Registered : Dec 12 13:43:12 2022 +0000 (1y 15w 3d ago)
NickServ: Last seen : (less than two weeks ago)
NickServ: User seen : (less than two weeks ago)
NickServ: Flags : HideMail, Private
NickServ: jiatan has enabled nick protection
NickServ: *** End of Info ***

If we ignore the malicious actor for a bit - the original maintainer was caught at a bad time who happened to be on holiday when this went down. He responded on March 30, just a day after the public disclosure of the malware.

Thank you. None of these patches are urgent. I'm on a holiday and only
happened to look at my emails and it seems to be a major mess.

My proper investigation efforts likely start in the first days of
April. That is, I currently know only a few facts which alone are bad
enough.

Info will be updated here: https://tukaani.org/xz-backdoor/
[source]

They seem to be acting swift in removing Jia Tan's access and working towards recovery on the xz project. They've already started in reverting of some of the malicious commits.

On that same idea of reverting - Jia Tan before having their GitHub account suspended was also rewriting branches on an unrelated zstd repository. Seems a bit odd all things considered - perhaps they knew it was time to burn evidence. Unfortunately a few of the xz repositories were also disabled by GitHub making research a tad bit more annoying.

Access to this repository has been disabled by GitHub Staff due to a violation of GitHub's terms of service. If you are the owner of the repository, you may reach out to GitHub Support for more information.

On the malware front it's been interesting watching the discoveries come out on how the malware operates. Filippo Valsorda has an interesting thread on Blue Sky that broke down the malware roughly working as:

• Extracting payload from n (public key) during authentication from a certificate.
• Checking if the payload matches signature from an expected attacker key.
• If not - goes to regular behavior.
• If matches - executes the payload.

Very much appears to be intended to be malicious and difficult to detect as you'd need the private key of the attacker to even have a valid payload. Everything else would fail and thus skip the malware. Depending on the user running sshd (ie root) this might be a powerful remote code execution on infected systems running at the highest trust level. It just by accident made SSH inefficient and thus noticeable by Andres during an investigation.

This vector of assumed attack is unlike anything I've heard of before - it's crazy to think about. Imagine the infected SSH agent sitting there ready to execute remote code, but only if it gets a payload embedded in a certificate that is signed by the key its expecting. If it does - its ready to execute, hide its tracks and prevent logging information - since it's technically not logging in via the regular SSH process. Any other key connecting like normal or via scripting would just fail the validation and skip the malware.

Some folks have even discovered the "kill switch" embedded in the malware that apparently disables its functionality. A seemingly random generated key/value pair that the malware reacts to by not hooking its malicious logic.

yolAbejyiejuvnup=Evjtgvsh5okmkAvj

This must have been invented so the malware author(s) could have a clean copy of SSH while working on a exploited one.

Another party quickly worked to develop a POC for the exploit - proving exactly how it worked down to the generation of the command in the bogus certificate.

Finally a party works on understanding the bash obfuscation that led to the delivery of the malicious native objects into the link pipeline. Which is another piece of the puzzle that is elegant in nature with head tricks, file carving, substitution ciphers, RC4 encryption and more.

Everything about this situation feels like more than one talented individual can do. My gut says there is a lot more behind these actions than one person as someone with an amazing open source set of knowledge, immense obfuscation skills for bash and malware seems like a rare person. It reminded me of that YouTube video of a previous NSO engineer explaining how many engineers work together when building a robust exploit.

It seems like a recurring pattern that we have chunks of our software world written, maintained and kept alive by unpaid volunteers. It seems an entitled vocal crowd screams louder than the calm and collected individuals that work together in the open space. This leads to burnout and unfortunate mistakes which I've blogged about many times.

Appendix