[Question] How to deal with 32bit binaries when doing a snapshot ?

[GuillaumeOrlando]

Hi,
First of all this project looks amazing :)

I am trying to replicate the example shown in "Fuzzing Modern UDP Game Protocols With Snapshot-based Fuzzers", but I have a question.

What if my target binary is a 32bit binary running on a 32bit Hyper-V virtual machine ?
I saw that the bdump.js script is looking for some 64bits registers for the "regs.json" output, and the small paragraph about WOW64 in the README.md confuses me.

Will I be able to run the fuzzer if a create a 32bit version of "bdump.js" ?

For the context, I am trying to fuzz a 32bit library used by a 32bit binary.
If I use a 32bit debuggee virtual machine and a 32bit version of WinDBG, everything is fine, I can break on the desired function and replicate "bdump.js"with my own script.

But if I use a 64bit debuggee virtual machine with whatever version of WinDbg, I am not able to see my target DLL anymore, so I can't break on it nor take a process snapshot ... All I see are some wow64 related DLL when I switch into the context of the target process.

I have tried to play with ".effmach" and "!wow64exts.sw" but my target DLL is never accessible that way when running on a 64bit platform.

Unfortunately, I can't find a lot of resources online regarding this issue. I can't even figure out if this is something possible.

The README of the project seems to imply that the snapshot must be done on a 64bit platform, but I'm not sure as this is not explicitly written.

I know that this question is not directly related to the project, and more about the "bdump.js" script usage / WinDbg usage, but since my goal is to fuzz this library using wtf, I take my chance here :)

Thank you

[tostercx]

But if I use a 64bit debuggee virtual machine with whatever version of WinDbg, I am not able to see my target DLL anymore, so I can't break on it nor take a process snapshot ... All I see are some wow64 related DLL when a switch into the context of the target process.

Did you reload the symbols? For some reason sometimes switching to the process isn't enough, switching to a specific thread does usually show them. Not sure if correct, but I used:

!process /m myprocess.exe 0 -1 // find processes, dump all info
.thread /r /p ffffcc87328e8080 // switch to the first thread + reload
lmsmu                          // check user modules

[GuillaumeOrlando]

Wow, thanks for your help.

I was using the following commands:

.process /i /p <proc_offset>
.reload /user

Which indeed, did not show the full list of loaded modules for an obscure reason.

After switching to the first thread, everything is there :)
Thank you

[tostercx]

No worries :)

I'm following the blogpost for a 32bit binary as well atm - for the record I did get the fuzzer working and I'm getting good coverage, so I think I dumped/set up everything correctly. But. Sometimes I'm getting really weird traces. Something like:

mymodule+0x45b69   // everything running normally
mymodule+0x45b6c
mymodule+0x45b6e   // call to a function in the same module - valid address and everything
nt!KiPageFault+0x0 // how?!
nt!KiPageFault+0x1
nt!KiPageFault+0x8

Tell me if you run in the same problem. I'll probably open a separate issue if I don't get to the bottom of it soon.

edit

I think I solved it, same as #21 - had to disable pagefiles + take a full dump. yrp604 really is the man! :)

[y0ny0ns0n]

I also encountered similar problem when I tried to take snapshot on 32bit program.

I just solved it reading IAT address to that DLL or vftable func on that DLL and calculate target function address manually. In my case, target DLL mapped on memory but just not showed on module list.

And if can't see the mapped DLL on memory, I suggest to take full snapshot after disable pagefile creation. I saw some binary with very big segment size just don't mapping whole binary at first time( maybe Copy-On-Write issue..? ).

If you suspect CoW is the cause of the problem, I think you could also try Reflective DLL injection. I didn't tried it yet for snapshot but I think there will be no CoW issue if you tried this way( of course, you have to use SetBreakpoint with fixed VA in this way ).

[0vercl0k]

No worries :)

I'm following the blogpost for a 32bit binary as well atm - for the record I did get the fuzzer working and I'm getting good coverage, so I think I dumped/set up everything correctly. But. Sometimes I'm getting really weird traces. Something like:

mymodule+0x45b69   // everything running normally
mymodule+0x45b6c
mymodule+0x45b6e   // call to a function in the same module - valid address and everything
nt!KiPageFault+0x0 // how?!
nt!KiPageFault+0x1
nt!KiPageFault+0x8

Tell me if you run in the same problem. I'll probably open a separate issue if I don't get to the bottom of it soon.

edit

I think I solved it, same as #21 - had to disable pagefiles + take a full dump. yrp604 really is the man! :)

One easy way to diagnose that class of problem is to open the dump in WinDbg and try to read the address. If there's no memory there it's bad news - it means you are going to take another snapshot; some prefer to set-up the VM w/ no pagefile and whatnot, some prefer to run lockmem on the target before snapshotting.

For the whole 32-bit thing, I personally haven't done any 32-bit fuzzing but have talked with people that have. Although it would be possible to fuzz a 32-bit kernel, wtf doesn't have support for that. So what people have done is to run the 32-bit app on a 64-bit kernel (WoW64) as you mentioned 😄

In any case, if you encounter any issues or think I should clarify the README I am happy to do so!

Cheers

[tostercx]

One easy way to diagnose that class of problem is to open the dump in WinDbg and try to read the address. If there's no memory there it's bad news - it means you are going to take another snapshot; some prefer to set-up the VM w/ no pagefile and whatnot, some prefer to run lockmem on the target before snapshotting.

Thanks (and thanks for the awesome project)! Turning off pagefiles solved most of the cases (had the same 2 byte skip pattern further down the trace) and setting up lockmem got rid of the few remaining ones :) Didn't check mem for the first ones but there was no memory loaded there for the latter ones (windbg showed ??'s).

In any case, if you encounter any issues or think I should clarify the README I am happy to do so!

Everything's running smoothly now. Lockmem / pagefile / kva stuff might be worth a mention in the readme as I see there are ~4 related issues to that.

Also, if anyone needs it I hacked together an ugly script to load wow64 traces in tenet for 32bit targets (tho not sure if it's 100% correct): gaasedelen/tenet#13

[0vercl0k]

Awesome! I filed #86 to try to address that, I might tag you when I make the changes to have your opinion on them if that's ok :)

Cheers

[0vercl0k]

@GuillaumeOrlando let me know if you have any other questions - I'll close the issue if not :)

Thanks for trying wtf y'all 🙏🏽🙏🏽

Cheers