Mutual Members
During the several times that they had met, the Avengers and X-Men seemed to have established a genuine respect for each other. Though the occasional distrust might cause friction, so far they always parted as allies and went their separate ways. However, as the only times the teams associated was during times of crisis (aside from attending the wedding of Reed Richards and Sue Storm, which was hardly uneventful), their interaction was as brief as it was infrequent. Despite this, a more personal, as well as personable, face was put on the X-Men when a large number of Avengers decided to leave at one time, causing the ranks of the Avengers to drop by nearly half. Announcing a membership drive, the Avengers accept the application of the former X-Man known as the Beast, giving him probationary and, soon thereafter, full membership. [Avengers (1st series) #137, 151]
Over the months that followed, the Beast achieved a level of admiration by the public he had never and could never receive as an X-Man. So respected was he that he was among the candidates selected for membership when the ranks of the team was to be hand-picked by Henry Peter Gyrich, the team’s government liaison. [Avengers (1st series) #181] During his tenure as an Avenger, the Beast formed friendships and a reputation among his peers which would last far beyond his time with the team. In a way, he eventually became a permanent link between the two groups. Each side would always have at least one person that they trusted implicitly in their dealings.
Luckily for the Beast, however, the Avengers never learned of the one moment where he ultimately proved where his heart lay. While on monitor duty at Avengers mansion, the Beast received an alert by local police that the X-Men were attacking the Hellfire Club, an exclusive club that seemed legitimate to the public, but in fact was run by a group of nefarious mutants, bent on the acquisition of wealth and power. Rather than alert the rest of the team, the Beast erased the alert and went off to aid his former team. [X-Men (1st series) #134-138]
The next encounter between the X-Men and the Avengers, while of being more indirect in nature, dealt with their mutual failings with a common ally. Carol Danvers, the former Avenger known as Ms. Marvel, had done battle with and had become the perennial foe of Rogue, one of the members of the latest incarnation of the Brotherhood of Evil Mutants. In a climactic battle on the Golden Gate Bridge, Rogue inadvertently permanently absorbed the powers and even the personality of Ms. Marvel, nearly killing her in the process. [Marvel Super-Heroes (2nd series) #11]
Rather than seek the aid of the Avengers, whom she felt had betrayed her with their thoughtlessness from months before, Ms. Marvel sought out the X-Men and enlisted the aid of Professor X in restoring her mind. 
Eventually learning what had happened with Rogue, the Avengers traveled to X-Men’s estate, where Xavier and his team gave Carol and the Avengers privacy for their bitter reunion. The fact that she felt such hard feelings toward them surprised the Avengers, as they had not realized the circumstances of Carol’s departure from their group had not been as it had seemed. Though the two teams did not exactly encounter each other during this event, it could not have been easy for the Avengers to have been so rebuked on the X-Men’s home turf. [Avengers Annual #10]
Unfortunately for her, Carol would soon feel betrayed by the X-Men as well. Within a few weeks after her arrival, Rogue too came to Xavier’s school, seeking sanctuary. Having absorbed Carol’s psyche as well as powers, she pleaded with the X-Men for help and they agreed. Feeling now let down by the X-Men as much as she had been by the Avengers, Carol eventually accepted membership among the galaxy-faring Starjammers soon thereafter, leaving Earth altogether. In a way, it could be argued that Carol had become the antithesis of the Beast. While the Beast had successfully transitioned from being an X-Man to an Avenger, being accepted by both teams equally, Ms. Marvel had unintentionally been betrayed through negligence and thus rejected both. It would be many years before all was forgiven, and it took over a galaxy of distance to accomplish this. [Uncanny X-Men #171, 174]
The Magneto Problem
Up until this point, most of the conflict between the two teams had been isolated moments of misunderstandings. For the most part, the Avengers clearly respected the X-Men, at least much more than the populace at large. However, around this time, the public’s distrust of mutants had grown - and the actions of the X-Men had not helped how people viewed them.
First, the very villain who had been responsible for Ms. Marvel’s condition, the mutant known as Rogue, had been taken in by the X-Men. Though the Avengers had also previously taken in former mutant villains, Rogue’s actions had drastically affected one of their own. This would cause of certain amount of distrust when the teams next met, which was on an alien world created by the nigh-omnipotent entity known as the Beyonder.
Along with several unaligned heroes, the X-Men and Avengers were abducted by the cosmic entity calling itself the Beyonder to fight in a bizarre, yet violent experiment on an alien world in 
another galaxy. Initially chosen by the Beyonder as representatives of “good” who would then fight another group also chosen by the Beyonder to represent “evil,” their years-old alliance seemed to be on shaky ground. The impetus for this change was the presence of Magneto, whom the Beyonder had chosen to appear with the heroes. When the X-Men chose to defend Magneto’s presence, saying that he had changed, many of the Avengers were wary of the team’s motives. [Marvel Super-Heroes Secret Wars #1]
So great was the distrust that arose that Xavier soon after decided for he and his X-Men to take their leave of the headquarters commandeered by the 
Avengers’ group and go in search of the fortress of Magneto, who had left their number shortly after their arrival on the planet. While not turning their backs on the Avengers, the X-Men had decided to act as a force independent of Captain America’s. Some days later, the X-Men displayed their allegiance by offering to cover the Avengers’ vigilance against Galactus consuming the planet, while the Avengers went in search of the She-Hulk, who had decided to attack the villains’ headquarters on her own. The former mistrust not even an issue in his response, Captain America accepted immediately. [Marvel Super-Heroes Secret Wars #7]
Despite the Avengers’ accepting the X-Men’s offer and their continued alliance during the remainder of the Secret War, the Avengers’ opinion of the X-Men had now been clouded. While they may not have announced so publicly, they did so behind closed doors when they so informed a former member of the X-Men after she literally resurfaced after her apparent death. Having been believed dead for months, the Avengers were startled to discover Jean Grey very much alive, having been restored to health by a bizarre set of circumstances orchestrated by the Phoenix Force. Waking to a world much harsher to mutants than when she had apparently died, Jean was shocked to be informed by Captain America that the X-Men were associating with Magneto. Cap even went so far as recommending that it would be unwise for Jean to seek out Xavier and inform him of her return. [Fantastic Four (1st series) #286]
A few months afterward, the Avengers were called in by local authorities to specifically deal with Magneto. Ultimately, they were thwarted by another mutant - not one of the X-Men, but one of their wards, a member of the New Mutants team. A few weeks previously, the Beyonder had journeyed to Earth itself from beyond and, in another of his experiments, killed the New Mutants, only to resurrect them shortly afterwards. Distraught at their resulting mental trauma, Magneto - who had recently become headmaster of the school - allowed them to be turned over to Emma Frost and her Massachusetts Academy, only later to realize that he did this as a result of mental coercion by the Hellion called Empath.
Realizing that Magneto was on the way to recover them, Emma Frost had a local sheriff contact the Avengers, claiming that Magneto intended to abduct one of the students. When the Avengers intercepted, Magneto was fought to a standstill and eventually surrendered, rather than allow one of his wards to be harmed. However, before he could be taken into custody, he was teleported away by the New Mutant known as Magik, sister of the X-Men’s Colossus. Though they were fully aware of where Magneto and the New Mutants were based, the Avengers seemed to decline to pursue, possibly demonstrating their ambivalence and uncertainty of their outlook to the X-Men. [New Mutants (1st series) #40]
However, this was not the last time the Avengers would act to bring Magneto to trial. When pieces of Magneto former orbital facility, Asteroid M, fell to Earth, the Avengers knew that Magneto would go looking for it and set a trap. Just before it was sprung, however, the X-Men 
arrived to thwart the arrest. Despite the fact that they were helping a known mass murderer resist arrest, even Captain America refused to paint the X-Men with the same brush of villainy as Magneto. Luckily, before the situation went too far, Magneto surrendered to the Avengers to face his crimes to an international court. Much to everyone’s surprise, however, Magneto was freed on a technicality, enabling his association with the X-Men to continue without the mystique of harboring a “known criminal.” [X-Men vs. Avengers #1-4]
Though no longer harboring a fugitive, the X-Men’s association with Magneto was not long to last. Some months later, the X-Men were drawn into a conflict against a mystical foe called the Adversary in Dallas. Eventually learning the price of victory, the X-Men willingly sacrificed their lives to become the energy weapon the shamanic mutant known as Forge would need to end his threat. Though they were in fact saved from oblivion by Roma, the Omniversal Majestryx, in an attempt to maintain cosmic balance, the rest of the world believed the X-Men were dead. Though the public did not seem to acknowledge the sacrifice, some even going so far as to say that they had “gotten what they deserved,” no doubt this more than sealed their status as heroes to the Avengers. [Uncanny X-Men #227]