If you want to embed videos in your muse site, without using a service like youtube or vimeo, then you will need to manually upload the video files on the server and use them via the video tag available in html5, HTML5 Video
To insert the video tags in your muse pages, you will need to use the 'Object -> Insert HTML' option.
Since this option is not available in Muse, out of the box, I will recommend that you post this on our ideas section over here, https://forums.adobe.com/community/muse/ideas, and let our devs team know of this requirement.
- Abhishek Maurya