The assembly of dusty galaxies at $z \geq 4$: the build-up of stellar mass and its scaling relations with hints from early JWST data