Update: That Zman Biur guy sure knows his stuff! Yes, it is a caper bush. In Hebrew it is called Tzalaf or Nitzpah (see Demai 1:1).
The caper bush is mentioned several times in Talmudic literature. For instance there is a discussion of what blessing to recite before eating different parts of the tree (Berachot 36). Our sages say that the caper bush is the boldest tree (Beitzah 25B). Rashi comments there that he doesn't know what is bold about the Tzalaf. Tosefot says that its boldness is in that three different parts of the tree can be eaten and that it provides new fruit every day. In Rabbi Rafael Halperin's Encyclopedia LeBeit Yisrael it is written that the caper bush's boldness is in that it is hard to destroy: If you burn it or chop it down it will quickly regenerate. I remember Rabbi David Samson telling me many years ago that the Tzalaf is bold in that it can take root in nooks and cranies, something that most trees cannot do.
In the Jerusalem Talmud (Shabbat ch. 15 halacha 3) there is a story where our caper bush plays a major role:
Once a pious man took a walk on the Sabbath in his vineyard and he saw a breach (in the fence that surrounded the vineyard). He thought to himself how he was going to repair it when after Sabbath. (Then) he said, "Since I thought to repair the fence (during the Sabbath, when he was usually careful not to think about mundane matters), I will never repair it."(This he did to atone for thinking about this on the Sabbath). What did the Holy One Blessed Be He do? He caused a caper bush to grow there and it filled the breach. From this caper bush he ate and it provided him with sustenance all his life.
Here is another caper bush in bloom in the Jerusalem neighborhood of Givat Shaul: