I am currently writing a story with two boys and a girl as the main character. I really want the two boys (brothers) to really hate each other, at least in the beginning. I have one scene where they actually start this violent brawl, but everything after that kinda makes it look like they just annoy each other. Eventually they end up liking each other.

The girl needs to have this quiet personality, and I want her to be a bit afraid of other people because of her bad history with her brother. I feel like I'm making her too comfortable with the brothers. She is supposed to feel more comfortable around them, but just a little bit until later on.

Does anyone have any ideas on how to make the hatred between the brothers seem real and not just mutual annoyance, and making the girl seem more timid and introverted?

Coming to this very late, but it might be useful to someone...

There are two issues in the original question:  one is how to make characters consistent with the original intent, and the other is how to handle characters who change.

Making characters feel consistent to the reader means giving them sufficient depth at the start so that their core personalities are clear, and the causes of (in this case) antagonism are both clear and permanent.   It's typical in real life for relatives to go through up and down periods--each will have trigger points, hot buttons, which when stepped on will flare up into anger and/or hatred, as old grievances are suddenly released....and then things may quiet back down again for awhile, when that release has slackened the tension between them.  It's important to fully understand why the brothers are near-enemies as well as near-relatives...to have reasons readers can believe in.   Older siblings often have, or claim, authority over younger ones, who resent it.  They may bully the younger ones (and younger siblings do not forget these things, even the most trivial.)    Parents or other significant adults may have favored one over the other, blamed one for something the other did (or that someone else did.)  One may have been "the good son" and the other "the bad son" when the actual difference between them wasn't that great...or one was "the musician--he's going to be famous one day" and the other "Oh, he's fine; he'll take over the store after me" (when that one is actually desperate to go to college and study physics--but the musician son is favored and all the money's spent on him.)   It's the depth of characterization--the writer's clear understanding of those characters' innate and acquired characteristics and thus their motivations--that creates consistency in the characters.  The history doesn't change--having  a sibling hit the same hot button repeatedly will evoke the same flare of anger.

But handling character change is essential, however a conflict between them goes.  Characters who don't change at all can be useful in some situations, but most effective characters do change in response to what's going on, just as people do.  They may change into more rigidity, but they change.    If characters start changing--insist on change--when you don't want them to, it's often a sign that the story as first imagined was too flat.  Let them change and see what happens.  The core of the personality won't change--the backstory of the character won't change--but what the character does with those raw materials can change.   That's why any two writers being given the same characters to use will write very different stories.