The article is devoted to the study of methods for detecting malicious software (malware) using static-dynamic analysis. A method for detecting malware is proposed, in which the number of parameters of the behavior of executable files is optimized using clustering of insignificant features, and also fuzzy-hashing of own functions is used when constructing a call trace. A prototype of a malware detection system based on the proposed method has been developed. Experimental studies assess the effectiveness of the proposed method. The efficiency of malware detection by the developed prototype is estimated. According to the verification results, the developed prototype was able to improve the detection efficiency of malware.