If you have to concern yourself with criticism from others, you should consider the opinion of a person with clear eyes1.
When I was in China, Master Nyojo of Tendo Monastery chose me as his personal attendant saying, “Although Dogen is a foreigner, he is a man of capacity.” I declined it unequivocally.
I refused knowing it was important for establishing my reputation in Japan and for the sake of practicing the Way. This is because I thought there might have been someone with clear eyes in the assembly who was critical of a foreigner becoming the abbot’s attendant in such a great monastery, implying there were no men of ability in the great Song China. I had to be very careful. I wrote what I thought in a letter to the abbot. When Master Nyojo read it, he understood my respect for his country and my feeling of shame before people having clear eyes, so he did not ask me again.