2hrs training rides won't cut it if your races are 3-4hrs long. Anyone that tells you to "just ride harder" for short rides, doesn't know anything about exercise physiology. Endurance fitness is like a building. Needs a foundation before you can add walls, roof, and all the important stuff that makes it function. Focusing on very short rides, zone 4-5 intervals guarantees you'll be great for about 20 minutes into your race.
Two equal riders, same genetic ability. Rider #1 does 5hrs a week, never riding more than 2hrs, focusing on zone 4 & 5 efforts every ride, hardly ever riding in zone 2. Same things for 3 months. Rider #2 does 2 months of zone 2, 7 hrs a week. One 4hr ride, two 1.5hrs rides. Each ride two days apart. Nothing above zone 2 for those two months. Third month our rider #2 does exactly what rider #1 has been doing for the last two weeks. Rider #2 will smoke rider #1. Endurance training 101.
My background;, a few national titles, Crusher, RPI, BWR, DK200 finisher, medaled at worlds. This not to toot the horn, just to clarify that my opinion comes from direct personal experience of racing at the national level for 20 years. Not something I read in a forum. 19hrs/week for me so far in '20, the bulk of it in zone 2.
Human body stores about 2hrs worth of glycogen, regardless of training. At LT2 or zone 3 (gravel race pace), you will deplete at 2hrs. Key to gaining both the mitochondrial density required for peak aerobic efficiency and best possible LT2 power output, is doing ride that last longer than 2hrs. Eat during the ride and try to have steady energy level for the whole ride. Know that even the top pros don't ride at zone 4 for a 4hrs race. Body can't do that. They'll be in zone 3, (above threshold) with short periods of zone 4. A less trained athlete has to plan on doing most of that 4hrs in Zone 2, specifically below LT2 with just short periods of zone 3.
Basically, you won't have the efficiency nor capacity to survive the lactic build up, ATP depletion of long Z3 efforts without blowing up, so don't. Your goal, any gravel racers goal is make S*#t tons of LT2 power. Don't be dismayed. That's the exertion level right before you start making more lactic acid than your system can clear. So you slowly get that burn that is unsustainable. Learn to ride as close to that threshold as possible without going over. If you do go over in training, don't stop whatever you do. Learn to "roll off the throttle" just a little. Reduce your effort by the smallest amount you can, grit your teeth while the lactic subsides and you again reach a sustainable pace that isn't lighting your legs on fire. Eat, drink, Lt2. That, in a nutshell, is gravel racing.
TL

R version, yes you do have to do 3-4hr training rides if you want to do well in 4hr races.
HTH