Why didn't Rav Soloveitchik make aliyah?

This week, I came across a remarkable letter, written by Rabbi Yosef Dov Soloveitchik shortly after his Rebbetzin's passing, in 1967. An Israeli schoolteacher who had just taught her students Rav Soloveitchik's inspirational writings on Zionism wrote to him, to ask why he had not actualized his ideals by moving to Israel, and he responded most movingly.

The text of the Hebrew letter is found on Page רנד of מפניני הרב, as well as toward the end of an article available here. Here is my attempt at an English translation:

17 Tammuz 5767 (1967)
Her honour, Madam Miriam Shiloh, שתחיה,

Shalom uBerachah! I request forgiveness for not addressing her letter immediately, and delaying until now. It was difficult for me, and it remains difficult, to write. I am immersed in mourning for the death of my beloved wife, the pupil of my eye and beloved of my soul, of blessed memory. My world has collapsed and is ruined. Even now, I write with tears and with the blood of my torn and agitated heart.

I thank her for her words, and I accept her rebuke willingly. In truth, I have sinned against the Holy Land. I am among those who have been slow. Of course, many causes, which do not depend upon me, have delayed me – but I do not seek an excuse, and I do not justify myself. I have sinned, and the [prisoner's] chain hangs from my neck.

Last year we decided, my wife z"l and I, to come to the Land and to spend six months there, to see the land and the nation dwelling there. But many thoughts are in the heart of man, and what happened happened, and my sighs are many and my heart grieves!

Many of my acquaintances and friends who dwell in the Land and in the Diaspora encourage me to come now. Madam also emphasized in her letter that it is now a propitious time as well as a time of spiritual crisis. But our sages tell us of an "Arch of Accounting" which was situated outside of Jerusalem, because one who takes accounts and finds his account lacking is upset, and regarding Jerusalem it is written, "the joy of the entire earth". How much more so for the "Arch of Black Mourning" in which I am currently found, that it must be outside Jerusalem. How could I now ascend to Jerusalem, when I am a man of a harsh and bitter spirit? G-d is not manifest when one is depressed, upset and pained.

Madam need not justify her words. I, too, am a simple Jew. In the language of the Sages of Yavneh (Berachot 17), I would say: I am a teacher, and many work with the community in the field of Torah education, as teachers. My work is in the large city of New York, in an institution with many students, and their work is in Givat Washington or in another community in a small institution. Whether one does much or little, so long as one directs his heart toward Heaven. The work all of us do is dedicated to nurturing the values of Judaism, the ancestral tradition and the Torah of our masters, authors of the tradition and its sages.

With great respect,
Yosef Dov haLevi Soloveitchik

To this I can add nothing...