As long as you don’t add the invalid paths to the graph, then that’s what Dijkstra does - finds the shortest path.
Suppose the farms are as above. When constructing the graph, you test the two shortest paths from 1 to 2 and discover that 3 is in between for the right-then-down path, and 4 is in between for the down-then-right path, so you don’t add either to the graph.
The trick is to also add additional vertices around each farm in each of the four main cardinal directions. So for farm 3, add vertices like this. (I’ve shown the vertex to the south below, but this shouldn’t be added to the graph because it’s the same location as farm 4 in my example.)
So, when you need to get from farm 1 to farm 2, the direct paths are both blocked and not in the graph, but the indirect paths via the extra vertices around farms 3 and 4 are available. And of course, the Dijkstra method should only visit a vertex representing an actual farm if it is the next farm in the sequence.
Does that help at all?
Note: I’ve not solved this, so I might have it all wrong. Edit: now I have.