I have a few Pi 3s so I'd figured I give RetroPie a shot. This will be less of a how to, and more of a review/comparison to modded Xboxes.
What you need
This is probably the biggest draw back of the Pi vs Xbox. An Xbox can be had for $40 pretty reliably, or probably free from somewhere. They generally come with everything you need to play, including at least one controller.
The Pi costs $40 by itself. You then need a power adapter, a HDMI cable, a case, and an SD card. Then you'll need controllers, 2 SNES knock offs will cost $30, or you can use XBOX360 ones if you want wireless and joysticks, but that'll bump the cost up $70 more.
You might have some of this stuff, but I don't think it's fair to assume you do for the price calculations. So the cost is at least double an Xbox, probably more if you want joysticks.
I won't give a full install guide here, you can follow the official guide easily enough. However, I will say the install is far easier than an Xbox. Download the image, write it to the SD card, pop the card in the Pi, attach controllers, and power it on. It autodetects the controllers, and lets you configure them. FTP in (u: pi, p: raspberry) and put roms in the ~/RetroPie/roms/* folders. Restart and the systems where it finds roms will show up in the list.
There isn't much to config, you can pretty much just play out of the box, however, there were some things I wanted to change. For example, I found the layout of NES buttons to be annoying. A bit of research told me that while the initial global controller set up is easy, there is no way to edit per system controls without editing the cfg files. For NES the file is at /opt/retropie/configs/nes/retroarch.cfg
You can SSH into the Pi with ssh email@example.com (change the IP to yours), then cd to there, and use nano to edit that file. Alternatively, you can FTP the file back and forth. The config is very confusing due to the fact that you have the labels on the controller and in the file which may not match. The numbers refer to the position of the physical button on your controller. The letters refer to what that button should do. For the Buffalo SNES knock offs this was my config for NES:
input_player1_b_btn = 3
input_player1_a_btn = 1
input_player1_y_btn = 2
input_player1_x_btn = 0
I think it works pretty well. There are some rough edges, but I guess those are mainly a concern for someone like me that wants to configure everything. All the games I tested work well. The one exception was N64, where they stuttered quite a bit. The Xbox was never great at N64 either, but it feels like it did better than the Pi. Perhaps that's just the few random games I tested, or maybe because the Pi doesn't actually have a video card.
Also there is no easy way to turn the Pi on and off, you just unplug it. For what it's worth, I measured the Pi as using about 2 watts (compared to the Xbox at about 50 w), so leaving it turned on 24/7 isn't a bad idea.
I was kind of disappointed at how poor N64 performance was. If that were better I'd spend the money for a set of wireless controllers and be quite happy. Each release of the Pi see a pretty significant speed bump, but at the same time I don't know if that is going to actually help, I think it's more the software and lack of video card.