As in, "Let's stop for now and have a tea." Some may find the 4:00 time to be a curious time for a break. To understand it, you must understand the pacing of meal breaks during the British day. The British traditionally start their meals later than we would think and then eat something every two hours. Breakfast is eaten whenever their schedule allows, but note that the British start their business day at 9:00 rather than 8:00. About 11:00 in the morning folks take a mid-morning break called, appropriately, "elevenses." This is just like our coffee break but also might involve pastries, Danish, or biscuits. They then work until 1:00. Lunch hour is between one and two in the afternoon. They come back and work until 4:00, when they take their "Tea." This would probably include biscuits. The business closes at 6:00 and folks head home for the evening meal (variously called dinner or supper) which may not be consumed until 9:00.
Historically, it is to Anna, Duchess of Bedford (wife of the 7th Duke of Bedford), that we owe the custom of the afternoon tea. She had tea and cakes served at 5:00 pm because, she "had a sinking feeling" in the afternoon. 21 Encyclopaedia Britanica 862 (1954).