This paper analyzes threats characteristic of the application layer of modern IoT systems, as well as the potential of machine learning algorithms for ensuring its security. The study focuses on the analysis of this layer, since IoT management applications have a highly diverse attack surface. The paper reviews studies proposing machine-learning-based solutions for detecting and preventing threats affecting modern IoT applications. Since these applications also serve as a point of interaction between humans and IoT systems, the paper also addresses the use of machine learning to counter social engineering techniques targeting IoT users.