Well, if you want to find all integer coordinates in triangle (0, 0), (x, y), (x, 0) where both x and y are non-negative integers, then you can find it with
count = ((x + 1) * (y + 1) - (gcd(x, y) + 1)) / 2 + (gcd(x, y) + 1)
Let me explain why this
(x + 1) * (y + 1) are all coordinates in rectangle (0, 0), (x, 0), (x, y), (0, y)
gcd(x, y) + 1 is number of coordinates on line (0, 0), (x, y)
So, when you take all points in rectangle without points on line, you are getting twice as number of points in the triangle (of course, without points on line). This holds because when you divide rectangle by diagonal you are getting two same triangles. After you calculated number of points without points on line, just add it back and you will get the result.
I took a look at the problem (a nice one). In this case, when x = n, y is not always an integer value. I was thinking how to solve, but didn't find a way to solve that less specific case.
