This is what Sigmund Freud referred to as "the uncanny."
Uncanny things are that which are unfamiliar or frightening to us, that we experience when the line between our imagination and reality is blurred. Freud used the example of a wax figure -- an object that holds human qualities but is ultimately inhuman. A wax figure causes a psychological disconnect as we try to reconcile that which appears real with the objective knowledge that it is not real. Think of the term "uncanny valley," coined by robotics professor Masahiro Mori in 1970, which refers to robotic or 3D animated human figures that look and move almost -- but not perfectly -- like a real human being, inspiring a sense of revulsion and fear in the viewer.
Only God Forgives inspires that same sense of fear with its presentation of a story and characters who are uncanny. Like the work of David Cronenberg or David Lynch, Refn is able to create a world and characters that feel almost tangible, like a bizarro version of our own world that is, aesthetically, almost identical to our own -- almost. But there's a subtle quality to this setting and these characters that is "off." Something isn't right. Aesthetically, they are identical to real people, but in movement and behavior, there is an off-putting, violent, and fantastical quality. Their motivations are primitive and their actions unrestrained by social mores and societal conventions.
That idea of the uncanny extends to the relationship between the male and his mother. In "The Uncanny," Freud writes:
"Neurotic men declare that they feel there is something uncanny about the female genital organs. This unheimlich place, however, is the entrance to the former Heim [home] of all human beings, to the place where each one of us lived once upon a time and in the beginning. there is a joking saying that ‘Love is home-sickness’; and whenever a man dreams of a place or a country and says to himself, while he is still dreaming: 'this place is familiar to me, I’ve been here before,' we may interpret the place as being his mother’s genitals or her body. In this case too, then, the unheimlich is what was once heimisch, familiar; the prefix 'un' ['un-'] is the token of repression."
Heimlich, in German, means homely, and therefore "unheimlich" is the inverse -- that which is un-homely or, as Freud suggests, uncanny. And further, the uncanny is derived from that which is already familiar to us -- that which scares us is already a part of us, lurking in the darkest recesses of our minds. In the context of Only God Forgives, there is an incredible amount of understanding to be parsed from Freud's statement. Take, for instance, the first time we see Ryan Gosling's Julian interact with Mai: she ties him to a chair while she masturbates, and prophetic, dreamlike sequences are inter-cut with moments of reality. Those sequences are moments that are later repeated in the film as reality, inspiring the notion that perhaps Mai's vagina is prescient, which sounds like a novelty, but given the symbolic nature of the womb the end result is fascinating. Home is where we seek and find comfort, and if Freud is to be believed, men are constantly trying to find their way back home, to that familiar place that holds all the answers to their existence -- to get back to the womb.
Julian and his brother Billy's relationship with their mother is entirely Freudian in nature, from the basic castration complex (Crystal, their mother, taunts Julian by comparing the size of his penis to his brother's) to Freud's notion of the Oedipal complex, when it's insinuated that one or both sons had something to do with the murder of their father. Crystal's icy demeanor and overt manipulations denote a woman who seems to be conducting her own Freudian experiments on her sons. Though we are never given a detailed account of Julian's past, it's safe to assume from the dinner conversation he has with Crystal that she used maternal bias to emotionally exploit and control her children, and instilled in them notions of penile envy, placing a value on their genitals that created the aforementioned castration complex, which is a fear of emasculation in both a figurative and literal sense. Combine these psychological issues with a mother who is herself rather masculine, and the result is alarming. Billy (at least what little we know of him) is overtly masculine and aggressive, while Julian is quiet and sub-dominant. The two function as halves of a whole, the scales remaining infinitely imbalanced, until Billy dies, at which point the volatile masculine qualities that have been repressed in Julian both by himself and his mother exhibit a rubber band effect -- over his life, Julian has had to hold back these innate qualities for so long and so tightly, that when he inevitably releases them, they thrust forward with great, unstoppable force.
But his reactions aren't entirely the result of years of repression: with the presence of his mother comes that familiar idea of home, and as Freud posits, men will do anything to return to the womb, which they identify as the place they ultimately come from and belong. They either do this by seeking women who physically resemble their mothers, or, like Oedipus, committing patricide to eliminate the competition. Julian will do anything to gain his mother's favor, both because he was presumably neglected as a child due to his mother's preference for Billy, and because he desperately wants to return home. Unfortunately, what his mother demands is ultimately too much for Julian's moral compass, and he cannot bring himself to murder the man Crystal believes to be responsible for Billy's death. Unlike his mother, Julian is not compromised by grief-stricken notions of revenge, yet he continues to cow to his mother, which only serves to exacerbate his rage as he struggles to continue repressing the violent seed his mother planted with her neglect and manipulations.
The idea of the womb as home is further illustrated later in the film when Crystal is murdered by Chang, the Thai police officer who exacts his own personal brand of justice and sees himself as a God figure (Friedrich Nietzsche, a colleague of Freud, would find this theme agreeable to his own writings). When Julian finds his mother's limp body, he cuts her womb open and places his hands inside, an act which holds many connotations. First, that Julian is searching for home, and using the tactile interaction to feel his way back in. It also indicates that he's trying to see if home still feels the same as it used to, much in the same way we return to the house where we grew up to experience a lost feeling and engage a sense-memory, but often what we find is that, while the house looks familiar, it no longer smells or feels the same -- this leads us back to the notion of the uncanny. Entering a childhood home and finding it vacant or inhabited by new residents and their belongings forces us to reconcile that which we know to be real, tangible, and familiar, with that which is unfamiliar, and the unfamiliar, as Freud says, is frightening, but is also in and of itself familiar. Julian trying to re-enter his mother's womb -- or his home -- is an action that forces him to process the notion of the uncanny on two levels, simultaneously. He must accommodate the notion of his dead mother: aesthetically, she is almost the same, but, as in Masahiro Mori's definition of the uncanny valley, she is not perfect, or what she should be. At the same time, Julian returns to a home that is not what he left it or how he has remembered it in his idealization. When he places his hands in his mother's womb, there is a look of confusion and disappointment; this is not how it was supposed to be. Adding another layer of context to Julian's experience with the uncanny is Freud's hypothesis of womb-envy, the belief that men are not only trying to reconnect or return to the womb by any means necessary, but that they also envy women for their roles as primary nurturers because a womb indicates power -- power that a dominant male cannot possibly possess or acquire -- and has positive emotional connotations associated with the act of nurturing. Women are innately able to grow and care for another human being, which enables them to understand what it means to be selflessly kind and generous, while men are biologically lacking this function that unites two physical and emotional concepts -- the scientific and the abstract, or the objective and the subjective, combine to create that which is uncanny to men.
When Julian reaches into his mother's womb, in that instant, we are given complete understanding about this character, his relationship with his mother, and the psychology behind his actions. And when he ultimately submits himself to Chang to have his hands removed at the wrist, he submits to the idea of castration and to the idea that his masculinity was never his to begin with. He never once had control over his own masculinity due to his mother's emasculating behavior and his own repression in correlation to his upbringing. In the world of Only God Forgives, a man's hands are omnipotent. Chang sees his as a Godly source of justice, to keep the scales in balance, while Billy uses his hands for the devil's work, and Julian languishes somewhere in between the two. In "The Uncanny," Freud talks about Oedipus, who blinded himself after committing patricide (albeit unknowingly), correlating the fear of eye trauma (a juvenile fear) with fear of castration, as both organs are of significant importance and are often related in dreams, fantasy, and myth. Freud goes further, though, by suggesting that "Dismembered limbs, a severed head, a hand cut off at the wrist" are all forms and symbols of castration, and all "have something peculiarly uncanny about them."
If masculinity and power reside in the hands and fists of these men, then the removal of Julian's hands is an act of castration, and one to which he willingly submits. He cannot return home to his mother's womb, nor can he possess it, and he lacks the ability to seek such homely solace in the body of another due to his repressed masculinity, which has left him unable to connect or communicate with a woman in a way that is socially acceptable. He is incapable of replacing his brother or avenging his death, and as a result of his incompetence in all manners pertaining to his brother, his mother, and himself (particularly in recognizing that the frighteningly unfamiliar, or uncanny, is that which is familiar because it resides within him already -- i.e., the very same alarming violence of which Billy was capable), he volunteers castration of that which he now finds to be useless, although you could also read it as an act of sacrifice to prevent the repetition of sin. Either way, who better to emasculate you and punish your sins than a man who believes himself to be God -- especially when Julian has just lost the only God he ever knew: his mother.