Mysticism

Mysticism is the claim to a means of knowledge apart from, or superior to, reason and the evidence of the senses — revelation, intuition, instinct, or any form of "just knowing". Objectivism rejects it not primarily as false in its conclusions but as a rejection of the only method by which a conclusion could be established at all.

“Mysticism is the acceptance of allegations without evidence or proof, either apart from or against the evidence of one's senses and one's reason.”
~ Ayn Rand, "Faith and Force: The Destroyers of the Modern World", Philosophy: Who Needs It

The claim, and what is wrong with it

A mystic asserts something and, when asked how he knows, answers that the question does not apply — that the knowledge arrived by a route not open to inspection. The Objectivist objection is that this makes the assertion uncheckable in principle, and an uncheckable assertion carries no cognitive content:

There is no way to distinguish revelation from imagination. If a claim's warrant is unavailable to anyone else, it is also unavailable to the claimant. A feeling of certainty is a psychological state, not evidence about the world. See Emotions and Arbitrary.
Conflicting revelations cannot be adjudicated. Two mystics who disagree have no method of settling it, which is why the historical resolution has so often been force. See Initiation of Force and Statism.
It presupposes what it denies. A mystic uses concepts, language and inference — all products of the reason he is setting aside — to convey the claim. See Fallacy of the stolen concept.

Objectivism holds that a claim advanced without evidence is arbitrary: not false, which would still be a cognitive status, but outside cognition, to be dismissed rather than refuted. See Faith.

Why it is an ethical matter too

Rand's interest in mysticism was not confined to epistemology. Her argument was that a doctrine placing the source of truth outside a man's own mind necessarily places the source of value there too, and that the two follow one another:

If knowledge comes from a realm beyond this one, then this world — the world a man perceives, works in and lives in — is demoted to appearance, and the values it contains are demoted with it. See Life and Value.
If man's mind is not competent to know, he is not competent to run his own life, and someone or something else must. This is the root Rand traced from mysticism to altruism and collectivism.
The demand then made of him is sacrifice: the surrender of earthly values in exchange for a benefit that cannot be examined. Rand held that the unknowability of the beneficiary is not incidental but the point, since it makes the demand unlimited.

See Religion and Mind-Body Dichotomy.

The secular forms

Objectivism does not treat mysticism as coextensive with religion. What defines it is the appeal to a non-rational means of knowledge, and that appeal survives in avowedly secular doctrines:

Intuition or "gut feeling" offered as evidence rather than as a hypothesis to be checked.
The claim that some truths are accessible only to a particular race, class, sex or historical position — an epistemological privilege that cannot be examined from outside.
Appeal to an unanalysable "authentic self", "the will", or "instinct" as the arbiter of what is true or good. See Subjectivism.

Rand's term for the modern combination — a formally secular culture that has retained the mystic's method — was that reason had been surrendered without the surrender being noticed.

Common misunderstandings

That it means being religious. Many religious people hold their beliefs as conclusions and are prepared to argue for them. The charge is about method, not affiliation.
That it condemns emotion or imagination. Objectivism treats emotions as consequences of one's values and imagination as a normal function of a conceptual mind. What it rejects is either one used as a source of knowledge. See Reason-Emotion Dichotomy.
That rejecting it requires proving a negative. The arbitrary is dismissed rather than disproved. The burden rests on the claim, and where no evidence is offered there is nothing to answer.
That "mysticism" here means the contemplative tradition. The word is used in Rand's specific sense — a claimed non-sensory means of knowledge — not as a description of any particular practice.

Discussion

References

Rand, Ayn. "Faith and Force: The Destroyers of the Modern World", Philosophy: Who Needs It.
Rand, Ayn. "The Objectivist Ethics", The Virtue of Selfishness.
Peikoff, Leonard. Objectivism: The Philosophy of Ayn Rand, chapter 5.