First Recruits, First Failures
Their courses laid and their ambitions declared, both Xavier and Magneto began the process of recruiting others that would share in their vision. In both cases, they would each fail.
For Magneto, his first true recruit was an Israeli mutant named Astra, who had the ability to teleport across cosmic distances. With her aid, he would acquire technology from alien civilizations, which would automatically give him advantage over the contemporary human military forces. 
Astra would also aid in spying on new recruits for Magneto’s Brotherhood, which would be the initial foot soldiers in his mutant army. [X-Men Forever #4]
Magneto’s first recruit, however, did not share Magneto’s goals - merely wanting riches, not power. After a short while, Astra departed Magneto’s employ. She would not be the last recruit who would leave, failing to be converted to Magneto’s new faith. In many instances, it seemed, the reason for Magneto failing to keep disciples was his harsh temper and extreme outlook. He knew best and he was the most powerful. While many would flock to his self-assuredness, they would leave after they discovered that that was all that he could offer in return.
As is sometimes expected in life, for every few steps forward, there is at least one back. Much like Magneto’s Astra, Xavier’s first recruit of Amelia Voght, another mutant with whom he had fallen in love and dated for some time, also had a crisis of conscience at Xavier’s plans. As his enlistment of disciples coalesced into reality, Amelia realized that Xavier was beginning a cycle of violence, which would lead to the war he was trying to avoid. The therapy Xavier had given the young Jean Grey was one thing - but the night Xavier’s first X-Man - the orphaned Scott Summers - came to live at the estate, it was too much for Amelia and she left. [Uncanny X-Men #309]
Disciples of their Cause
For those driven with a vision that cannot be contained, they find themselves compelled to share their dream with all who will listen. Those that chose to follow sometimes are converts, who can easily become either true believers… or fanatics. Sometimes some follow because they have nowhere else to go. Xavier and Magneto would have groups that fit into all categories.
As part of his vision, Charles Xavier converted his family estate to a place of learning, known publicly as “Xavier’s School for Gifted Youngster,” an exclusive private school. As the reputation for Xavier’s restrictive enrollment grew, Xavier found himself turning away students, wishing to matriculate at the elite institute. Ironically, Xavier would have to turn these students away, for being “different” than those whom he invited: mutants.
Those whom Xavier did enroll became his first X-Men. Xavier taught them all the things a normal student would learn - and then what they would need to know in their second career as mutant heroes. With a cold and stern demeanor, Xavier demanded complete obedience, most likely not too dissimilar from his own career in the military. Like a drill sergeant, Xavier molded the five teens into an effective fighting force - for he knew conflict was just over the horizon.
Xavier’s idea was multifold. First, he would train young mutants to use their powers responsibly, proving that they could be productive citizens of society. Secondly, they would be a force by which he could battle against evil mutants, which would turn the world against them. In the years since, it is clear that he intended these pupils to also be the custodians to the next generation. Nearly a decade later, Xavier would even step down as headmaster of his school, by then called the Xavier Institute for Higher Learning, and appoint his prime pupil, Jean Grey-Summers, ironically his first pupil, in his place. Subsequent to her death, another student, Scott Summers, would take up the role. At his side would be Emma Frost, Scott’s lover, who had been at odds with Xavier’s dream - but who was successfully converted to his dogma.
Meanwhile, Magneto found that his ability to inspire was lacking. Like Xavier, Magneto managed to put together a group of followers, which he called the Brotherhood of Evil Mutants. Also like Xavier, Magneto commanded strict obedience - however, that was all he offered. There was no kindness nor gestures of encouragement, which could have lead to the undying devotion that Xavier’s followers would lay upon him. Indeed, Magneto continually showed contempt to his followers, declaring that they were fools and ants before his might. Only he, Magneto, mattered. Not they.
As a result, Magneto was unable to expand his power base, continually being refused when he asked - or in most cases commanded - mutants that he discovered join his group. Eventually, two of his first recruits, Quicksilver and the Scarlet Witch, who only joined him because they felt they owed him a debt, left his Brotherhood. Soon after, they not only forswore Magneto’s cause, but joined the mighty Avengers, becoming examples of the possibility of human/mutant relations - examples that Xavier’s way was correct.