The orbital refinery is somewhat necessary for lunar mining. But lunar mining isn't necessary for the orbital refinery. The answer, therefore, is asteroids.
To go to the Moon with existing spacecraft, requires a means delivering rocket fuel into space. Either depots or docking of spacecraft.
In terms of lunar operations a critical location for docking or re-fueling is high earth orbit: GTO, GEO, L-points, or lunar orbit. And if one is also docking or re-fueling in LEO, one can use medium to small launchers to get to the Moon.
One could restrict it to only docking in LEO, and use existing hvy launchers- docking two payloads from two hvy launchers [though one could have multiple smaller launches and docking in LEO].
If one docking in some Lunar orbit and using hvy lift: hvy being any rocket that put about 5 of tonnes in lunar orbit. Any rocket that put 5 tons to GTO can put about 5 tons to Lunar orbit.
Existing Hvy lifts are Ariane V, Zenit, Proton, Altas V, Delta-IV.
Now, a dry weight vehicle of 5 tons is quite large when one consider that 90% mass could be rocket fuel- giving fueled mass of 50 tons- as much or more than Saturn V lifted to lunar orbit, so fuel depot is more flexible than simply docking to craft.
Fuel depot at lunar orbit or Earth/Moon L-1 essentially makes the existing hvy launch equal to Saturn V.
One could make a similar argument with depot at LEO- if have dry mass of 20 ton, it equal to fully fuel 200 ton craft [much bigger the Saturn V LEO payload.
Also one make the argument that fuel depot on the lunar surface gives a Saturn V type capability. Either of the three locations does this, and all three is giving a payload size to the moon [and returning from the moon] of around 3 times a Saturn V, to a number of existing rockets. Or about the same capability of Saturn V to medium launchers [even a greater number of existing rockets].
If you mining the Moon to make rocket fuel, this obviously is leading to fuel depot supplied from rocket fuel made of the Moon.
From a lunar miners point of view, all that is essential is a depot in L-1 with fuel on Moon being made on the Moon.
Or one start with rocket fuel delivered from Earth, to L-1 and lunar surface, and later as fuel is made on the Moon, lunar depot gets lunar made rocket fuel, followed by L-1 depot getting lunar made rocket fuel.
That combination gives greater than Saturn V capability. And lower cost because it allows the reusability of spacecraft landing and leaving the Moon.