You can choose to lock the Viewer zoom level, so that it doesn’t change when you switch between inputs of different sizes. Right-click on the Viewer, and select Prevent Auto Zoom to toggle between maintaining the same zoom level for all inputs (on) and changing it according to on-screen image dimensions (off). Alternatively, you can also press Alt+Z on Viewer, or toggle prevent auto zoom in the Viewer settings.