The Artificial Intelligence Act (AI Act) adopted by the European Union might serve as a global regulatory reference point. Heated negotiations over the AI Act have shown that reconciling the interests of numerous stakeholders is not an easy task. As well as creating clear and precise rules that would enable implementing effective safeguards for citizens against the manipulative potential of technology. The AI Act introduces the first legal definition of deep fakes and creates a system of protection against their harmful applications, that is based on transparency obligations. I argue that in the course of negotiations, the EU has made progress in regulating deep fakes, but the adopted solutions should only be an introduction to a more strict protection of individuals. The AI Act targets a specific group of deep fakes, leaving some of them unregulated or poorly regulated and disregarding the consequences of nonconsensual deep fake pornography. By analyzing the legal provisions related to deep fakes in the AI Act, I point to the rationale behind the chosen forms of protection, criticize the shortcomings and list the consequences of adopting the AI Act for deep fakes landscape.