Well, this was a fun one. I think I might have found an answer. I looked on some old OS maps in case the building was labelled. It wasn't, but a nearby building called Manor View was. It is still there and visible on Google Street View. A quick search of periodicals revealed that Mrs Ball (Manor View) let a house (Oak House) in 1904.
I wondered if this was a just a coincidence, despite the buildings (Manor View and Oak House Tea Room) being so close to each other, but found another article detailing the sale of Oak House in 1939. The address being 309 Handworth Road, which is the modern day Fish Bar.
So it seems that the building (coffee shop and fish bar) was a rather nice private residence (with a vinery!).