Question: Is it possible to write about a protagonist that people will not like, who dies in the end and about whom you should feel mixed emotions?Answer:
The straight answer is yes, but there are several ways of looking at this issue.
First, consider that the definition of the "protagonist" in Dramatica is the character who 1) Pursues the Story Goal and 2) Considers its importance. The word comes from Greek tragedy, and literally means the "first actor," because in Greek tragedy, the story begins with the protagonist setting out in pursuit of the goal.
Using this definition, you can find many stories in which the "villain" seems to take on the protagonist role. You may have a mad genius who embarks on a plan to take over the world (goal), and that's what begins the story. In such cases, the main character (the one through whose eyes the reader views the story), is actually the character who wants to Prevent having the goal achieved (and Considers the importance of that end).
You can also find examples of main characters who are also the protagonist, yet are not very nice people. Alex in A Clockwork Orange
commits numerous acts of violence, and there are other novels written from the point of view of selfish, sociopathic, or cruel characters. But even though readers may disapprove of these characters' actions, they may be intrigued by the character's perspective, charmed by the character's style of self-expression, or sympathetic to their situation. Alex, for example, can be seen as suffering unfairly despite his crimes.
As for having the protagonist die in the end, that is quite common in Tragedy or Comi-tragedy. Macbeth (another less-than-likeable character) fails in his ambition to found a line of kings because he becomes too ruthless. So while a reader may have some empathy for Macbeth in the beginning, his unfortunate end seems justified. That's also an example of "mixed emotions."
Keep in mind that what defines a Tragedy is that 1) the Story Goal is not achieved and 2) the main character ends up worse than when he started.