I have figured out the problem that was preventing the Yi Dashcam app from playing back videos on my Nexus 5, and found a workaround that allows playback.
I don't know how likely it is that others' problem are the same as mine, but here is how I got mine working.
As a reminder, the problem was that the app would connect to the camera, but would not play back videos, instead displaying the message:
Operation system does not support the resolution of the video file
Try another player?
However, downloading files from the camera worked fine, and playing back the downloaded videos with various playback apps worked fine as well.
My problem had to do with how Android (at least the version on a stock Nexus 5) handles WiFi networks that don't provide internet access, like, for example, the Wifi access point created by the Yi Dashcam.
In brief, before you hit "Connect" in the app, you need to provoke the operating system into showing you a message telling you your WiFi connection has no internet access, and you need to tell the OS to stay connected anyway, and to not ask again.
Here is the workaround in steps:
This assumes you have set up the dashcam's system ID and password and know what they both are.
1. Start the Yi Dashcam app. Do NOT press "Connect"
2. Using the Android Wifi settings controls, connect to the Dashcam's system ID and enter password if needed.
3. Wait 5-10 seconds - you should see a message appear at the top of the screen saying
"Wifi has no Internet access
Touch for options"
4. Touch the message. You should then see the the Dashcam's SSID and
"This network has no Internet access. Stay connected?"
5. Put a check mark in the box labelled "Don't ask again for this network"
6. Touch Yes.
7. Now go back to the Dashcam app and and hit Connect.
It should connect, and now playback of videos should work, assuming your problem was the same as mine.
Hope this helps someone.