Condensed answer: Ultegra rotors can be combined with 105 calipers when the rotors are of the correct size and rely on a mounting mechanism compatible with the hubs of the wheels.
Rotor Size
Disc brake rotors come in different sizes. In general, a larger rotor offers more braking power than a smaller one at the expense of extra weight.
Road bikes do not need the same stopping power as MTBs. Hence why MTB rotors are larger. For example, XC bikes usually have at least 160mm rotors whereas downhill rotors can reach 203mm in diameter.
Meanwhile, road bike rotors are either 140 or 160mm. The new Ultegra rotor should be of the correct size for the frame and fork that you have. If the bike is designed for 140mm rotors, but you try to install 160mm units on it, they won’t fit due to the lack of space. (Note: It’s possible to transition from 140mm to 160mm rotors with adapters depending on the disc brake mounts.)
If the bike is built for 160mm rotors, but you install 140mm on it, the new rotors will fit, but the brake pads will be too far to grab the entire brake track (the braking surface of the rotor). The outcome will be subpar braking.
Rotor Mounting System
The rotor mounts to the hub of the wheel. Ultegra rotors rely on Shimano’s patented technology known as Center Lock which secures the rotor to the hub via a lock ring similar to that on a cassette.
105 rotors use the same system. Thus, if you’re replacing a worn 105 model, the new Ultegra one will be compatible with the hub.
However, it’s possible that your bike uses a rotor from another brand (e.g., Tektro) that mounts to the hub via bolts. In that case, you will be unable to use an Ultegra rotor with your setup unless you replace the entire hub. Taking such drastic measures to install a specific rotor is not worth it because the entire wheel will have to be rebuilt.
FAQ: What are the advantages of using Ultegra rotors in this case?
Ultegra components are one step higher than 105 in Shimano’s hierarchy. That said, when it comes to rotors, the difference in quality isn’t all that perceivable. Besides, 105 components are of such good quality already that it would be naïve and inaccurate to say that an Ultegra rotor would make a perceivable difference.
Therefore, the main advantage would be availability. If you’re looking for a rotor, and an Ultegra one fits the criteria, it would be an acceptable option.