I found a couple of likely candidates but they were are published after I would have read this book around 1977. It kind of reminds me of Castles in the Attic. I can’t remember the title or the author. What I remember of the book I think this boy has to move in with his Aunt or Grandmother.
Could this be Knight’s Castle by Edward Eager?
Sounds a bit like Among the Dolls by William Sleator, though the main character is a girl (named Vicky).
Maybe Knight’s Castle by Edgar Eager? It’s about children visiting an Aunt and Uncle and cousins while their father is sick in the hospital. They shrink down and have adventures with both a toy castle and, in the end, a doll house.
Sorry, the author is Edward Eager, not Edgar.
I think it was published after 1977, but maybe try The Indian in the Cupboard by Lynne Reid Banks?
The Rackety Packety House by Fracis Hodgson Burnett; sounds like it might be the one.