RSA & bigint

[Ana06]

I have recently analysed a ransomware sample that uses the RSA encryption implementation from github.com/ezhangle/krypton (which is also similar to the implementation in github.com/bnoordhuis/mongrel2). I found tricky to identify it, until I found the 0x7FFF55AA = PERMANENT constant that pointed me in the right direction.

Issue to match RSA

So, I was wondering if we can create a capa rule for this type of RSA encryption. Characteristic code in the RSA_encrypt function:

• - 3 -> substruct 3 to calculate pads needed
  • assembly code: sub eax, 3
• out_data[0] = 0 -> to ensure encryption block is < modulus
  • assembly code: mov byte ptr [ecx], 0
• out_data[1] = 2-> I think this is the encryption flag (signing would be 1)
  • assembly code: mov byte ptr [edx+1], 2
• out_data[2+num_pads_needed] = 0; -> zero terminated
  • assembly code: mov byte ptr [edx+2], 0
• get_random_nonzero(&out_data[2], num_pads_needed) -> to randomize the encryption padding with non-zero bytes

The problem is that with the currently supported capa features we can only match offsets and numbers and in this case they are common numbers (0,1,2,3). Although matching then together inside a instruction feature could be enough to prevent false positives. @mandiant/capa-dev what do you think? Could this be enough?

The 0x7FFF55AA = PERMANENT constant is in a subfunction (likely several levels), so extending capa to add subfunction support would likely allows us to match this implementation in a more robust way. Is there already an issue in capa to support subfunction support? Could this be implemented and be useful for other cases as well?

Bigint

We could create capa rules for bigint functions, as identifying this functions may help reverse engineers identifying crypto. We can do this by creating a rule for every of the following three functions or a single generic rule for bigint functions. Note that the offset 8 and the 0x7FFF55AA = PERMANENT constant in the same instruction is shared by all functions.

@mandiant/capa-dev what do you think? Should I send a PR with tree rules (one per function) or a single rule that matches the three of them?

bi_copy

This function is called in bi_mod_power, called from RSA_public, called from RSA_encrypt
References:

• https://github.com/ezhangle/krypton/blob/147d69429bfb03cce7113dca6dba36e77f8a9325/src/bigint.c#L149
• https://github.com/bnoordhuis/mongrel2/blob/3e9b57d82aeb627be0aebfb346199bfdfd67e530/src/crypto/bigint.c#L162
  

bi_permanent

This function is called in bi_mod_power, called from RSA_public, called from RSA_encrypt. This function may also be called when initializing RSA.
References:

• https://github.com/ezhangle/krypton/blob/147d69429bfb03cce7113dca6dba36e77f8a9325/src/bigint.c#L161
• https://github.com/bnoordhuis/mongrel2/blob/3e9b57d82aeb627be0aebfb346199bfdfd67e530/src/crypto/bigint.c#L176
  

bi_depermanent

This function is called in bi_mod_power, called from RSA_public, called from RSA_encrypt.
References:

• https://github.com/ezhangle/krypton/blob/147d69429bfb03cce7113dca6dba36e77f8a9325/src/bigint.c#L177
• https://github.com/bnoordhuis/mongrel2/blob/3e9b57d82aeb627be0aebfb346199bfdfd67e530/src/crypto/bigint.c#L194
  

[mr-tz]

Awesome context here!

@mandiant/capa-dev what do you think? Could this be enough?

I'd say let's create the rule and run on the testfiles to see.

Is there already an issue in capa to support subfunction support? Could this be implemented and be useful for other cases as well?

See https://github.com/mandiant/capa/blob/master/doc/limitations.md#wrapper-functions-and-matches-in-child-functions

For the bigint rules I'd lean to one rule, but would ultimately trust your judgement to create 1 or 3.