Skip to main content

​
Encoders

Encoders have assisted with making payloads compatible with different processor architectures while at the same time helping with antivirus evasion. Encoders come into play with the role of changing the payload to run on different operating systems and architectures. These architectures include:
  • x64 — 64-bit Intel/AMD architecture
  • x86 — 32-bit Intel/AMD architecture
  • sparc — SPARC architecture
  • ppc — PowerPC architecture
  • mips — MIPS architecture
They are also needed to remove hexadecimal opcodes known as bad characters from the payload. Not only that but encoding the payload in different formats could help with the AV detection as mentioned above. However, the use of encoders strictly for AV evasion has diminished over time, as IPS/IDS manufacturers have improved how their protection software deals with signatures in malware and viruses. Shikata Ga Nai (SGN) was one of the most utilized encoding schemes back in the day because it was very hard to detect payloads encoded through its mechanism. However nowadays, modern detection methods have caught up, and these encoded payloads are far from being universally undetectable anymore. The name (仕方がない) means It cannot be helped or Nothing can be done about it, and rightfully so if we were reading this a few years ago. However, there are other methodologies we will explore to evade protection systems.

​
Selecting an Encoder

Before 2015, the Metasploit Framework had different submodules that took care of payloads and encoders. They were packed separately from the msfconsole script and were called msfpayload and msfencode. These two tools are located in /usr/share/framework2/. If we wanted to create our custom payload, we could do so through msfpayload, but we would have to encode it according to the target OS architecture using msfencode afterward. A pipe would take the output from one command and feed it into the next, which would generate an encoded payload, ready to be sent and run on the target machine.
After 2015, Metasploit consolidated these capabilities into msfvenom, which handles both payload generation and encoding. We will be talking about msfvenom in detail later on.

​
Generating Payload - Without Encoding

​
Generating Payload - With Encoding

We should now look at the first line of the $buf and see how it changes when applying an encoder like shikata_ga_nai. Notice how without encoding, the first line starts with \xda\xc1... but with 3 iterations of encoding, it starts with \xbb\x78.... This shows how the encoder modifies the payload structure.

​
Shikata Ga Nai Encoding

Shikata Ga Nai encoding visualization Suppose we want to select an encoder for an existing payload. Use the show encoders command to list compatible options for your current exploit and payload combination:
Suppose we want to select an encoder for an existing payload. Use the show encoders command to list compatible options for your current exploit and payload combination. In the previous example, we only see a few x64-compatible encoders. Like payloads, encoders are automatically filtered by exploit module. Different exploits expose different compatible encoders. For example:

​
Real-World Attack Scenario: Single Encoding

Let’s test msfvenom with a real Windows payload encoded once with SGN:
This creates an x86 Windows executable with a single SGN encoding pass. As you can see from the VirusTotal scan results below, single encoding is easily detected: VirusTotal detections for SGN encoded payload single iteration

​
Testing Multiple Iterations (More Payload Bloat, Still Detected)

Using the -i flag to apply multiple encoding passes:
Multiple iterations increase payload size but don’t guarantee AV evasion: VirusTotal detections for SGN encoded payload multiple iterations

​
Reality Check: Encoding Alone Is Not Enough

Both single and multiple iterations still trigger detection on most modern antivirus engines. Metasploit provides msf-virustotal for analyzing payloads (requires a free VirusTotal API key):

​
MSF - VirusTotal

As expected, most anti-virus products that we will encounter in the wild would still detect this payload so we would have to use other methods for AV evasion that are outside the scope of this module.