A stone wall.
Castles are generally either a large stone-walled tower or a group of buildings with a stone wall around them. The tower may be either rectangular or circular. The buildings may be made of stone or wood or any other building material.
If there are enough buildings to be a town, then it probably is not a castle. But not always. Some castles are small places where less than a dozen people could live. Others are huge estates within a wall.
No standard rule says when a building or group of buildings is a castle or not. It depends on the title given to the place by royalty or nobility and other factors.
For more on what exactly makes a place a castle, read “Castles” by Beth Smith or “Castle” by David MacAulay.