I need help understanding why the NEW_DOCUMENT event in LC is showing an incorrect hash for file that is created.
On 01APR I was looking at some suspicious events. A file (pure.exe) was downloaded via curl into the Users\Public folder (Screenshot 1). I went to the NEW_DOCUMENT event for that download and got the hash to put into VirusTotal; there was nothing found in VT. In LC I went to the folder to download the file so I could upload it to VT but the file wasn’t there. I was concerned the attacker had already deleted the file until I remembered that the system also had Cylance AV running. I check those logs and found that the file had been quarantined, but I noticed that the hash Cylance showed was different than LC. I thought maybe as the file was being quarantined that LC calculated the hash but that doesn’t seem to be the case. Unfortunately, I don’t have a screenshot from Cylance showing the hash.
On a test system, without Cylance, I downloaded the same file from the same domain and and again LC is showing the wrong hash for the file (Screenshot 2). The hash is actually different from the original events. Just to make sure this wasn’t some weird Windows issue I downloaded the file to a Linux host and the hash there matches my test system (Screenshot 3).
Am I mistaken in thinking that the hash value for a NEW_DOCUMENT event is the SHA256 for that file? Any help explaining why these values are different would be greatly appreciated.
Since I can only upload one screenshot, I got all three into one.
Hmm. Although there is a maximum size, if it is reached, we simply won’t hash the file at all, not just a subset. The limit is also significantly larger.
Will dig into this a little bit more tomorrow, but this is very weird, because clearly the hashing in general is not broken or it would have been extremely obvious before, so it’s really a question of figuring out what is special about this situation. Is there any way you could share the URL or a way to specifically reproduce the problem on our side?
The other thing that would be interesting to test is to do a file_hash of that specific file via the console to see if it also results in a bad hash.
Reviewing the details, I see it’s 30MB, I suspect what’s happening is that it takes a significant amount of time to download the file and we end up generating the hash before the file is done writing. It might seem like the logical thing to do is just wait until the file get Closed before computing the hash, but there’s actually some more subtleties going on because doing that would open up attacks where an attacker can write the malicious file to disk, not close it and therefore never generate a hash. We’ll take another look at what heuristic we use to decide when to compute it.
Thanks for that explanation. That’s an interesting edge case that I’m not familiar with, but in my opinion, the risk of providing an incorrect hash outweighs providing the correct hash a majority of the time. I could understand that choice if the hash from the original system and my test system matched, but the fact that they don’t makes the hash insignificant if it’s not reliably the same.
I do appreciate you considering options there might be to resolve this because I do really like LC.
In the mean time, I think it would be a good idea to add a note in the documentation for NEW_DOCUMENT that the hash might be incorrect if the file is not finished writing when it is calculated.
I think what may end up being the best is to split up the NEW_DOCUMENT event into two, one that is quick to let you know the file was created right when it was created, and another one with the hash when the file gets Closed (which might be minutes or hours).
IE: over 20Mb (stupid golang)