Extension Activity

The calculation

The two key assumptions made in using this method in this activity are that the Earth is a sphere with a radius of 6400 km.

To carry out the calculation we require the latitude and longitude values for the two places concerned.

Following the terminology used in the development of this formulation, latitudes (North) and longitudes (East) are given positive values, and latitudes (South) and longitudes (West) are given negative values.

The full calculation is given by

$$d=\frac{6400 \times \pi}{180} \times \cos^{-1} \big((\sin \theta \times \sin \phi ) + (\cos \theta \times \cos \phi \times \cos | c | )\big)$$

where $\theta$ and $\phi$ are the latitudes of the two places and $c$ is the absolute difference in longitudes of the two places.

To clearly demonstrate why the sign applied to an angle is important, we will break the following example calculation into two parts, finding the cos value ($D$) first using sine and cosine values rounded to four decimal places, and then finding the distance ($d$).

Example

Find the great circle distance between Paris (France) ($48.67^{\circ}$ N, $2.33^{\circ}$ E) and Melbourne ($37.82^{\circ}$ S, $144.97^{\circ}$ E) correct to the nearest kilometre.

Solution

By the definition given above, the Melbourne latitude value will be negative.

The absolute difference in longitude is $144.97^{\circ}$ - $2.33^{\circ}$ = $142.64^{\circ}$.

$\begin{aligned}[t] D &=\big((\sin \theta \times \sin \phi ) + (\cos \theta \times \cos \phi \times \cos | c | )\big)\\ &=\big((\sin 48.67^{\circ} \times \sin -37.82^{\circ} ) + (\cos 48.67^{\circ} \times \cos -37.82^{\circ} \times \cos 142.64^{\circ})\big)\\ &=\big((0.7509 \times -0.6132) + (0.6604 \times 0.7899 \times -0.7948) \big)\\ &=(-0.4605 - 0.4146)\\ &=-0.8571 \end{aligned}$ $$\text{ distance}=\frac{6400 \times \pi}{180} \times \cos^{-1}(-0.8571)=16642.623\ldots \approx 16643\text{ km}$$

We can see from the details of this calculation that the signs allocated to each latitude value have a major impact on the calculation of the angle between the radii leading to the two places. The negative value for cosine leads to an obtuse angle, fairly reflecting the distance between the two places chosen.

Example

Find the great circle distance between Paris (France) ($48.67^{\circ}$ N, $2.33^{\circ}$ E) and Melbourne ($37.82^{\circ}$ S, $144.97^{\circ}$ E) correct to the nearest kilometre.

Solution

By the definition given above, the Melbourne latitude value will be negative.

The absolute difference in longitude is $144.97^{\circ}$ - $2.33^{\circ}$ = $142.64^{\circ}$.

Performing the angle calculation in one operation with a graphic calculator we get,

$\begin{aligned}[t] &=\big((\sin 48.67^{\circ} \times \sin -37.82^{\circ} ) + (\cos 48.67^{\circ} \times \cos -37.82^{\circ} \times \cos | 142.64^{\circ})\big)\\ &=-0.857096106706\ldots \end{aligned}$

And hence finding the distance

$$\text{ distance}=\frac{6400 \times \pi}{180} \times \cos^{-1}(-0.857096106706)=16873.156\ldots \approx 16873\text{ km}$$

This calculation can be performed in one operation with a graphic calculator,

$$d=\frac{6400 \times \pi}{180} \times \cos^{-1} \big((\sin {48.67} \times \sin {-37.82} ) + (\cos {48.67} \times \cos {-37.82} \times \cos {142.64})\big)$$ $$= 16873.156\ldots$$ $$\approx 16873\text{ km}$$

We can see that using the full decimal storage capabilities of the calculator in the latter two calculations has resulted in a larger answer by some 240 km to that obtained where the sine and cosine values were rounded to four decimal places. This is one of the issues raised in a number of discussions regarding the merit of the various methods for calculating great circle distances - that using lesser accuracy of values in the calculation affects the cosine method more than the haversine or Vincenty's formulations.

To cope with this, we recommend performing the calculations as in the last two instances, using the calculator to store the full decimal values.

Example

Find the great circle distance between San Francisco (USA) ($37.783^{\circ}$ N, $122.417^{\circ}$ W) and Melbourne ($37.82^{\circ}$ S, $144.97^{\circ}$ E) correct to the nearest kilometre.

Solution

By the definition given above, the Melbourne latitude value and the San Francisco longitude value will be negative.

The absolute difference in longitude is $144.97^{\circ}$ - $(-122.417^{\circ})$ = $267.387^{\circ}$.

Breaking the calculation into two parts, finding the cos value ($D$) first, and then finding the distance ($d$).

\begin{align*} D &=\big((\sin \theta \times \sin \phi ) + (\cos \theta \times \cos \phi \times \cos | c | )\big)\\ &=\big((\sin 37.783^{\circ} \times \sin -37.82^{\circ} ) + (\cos 37.783^{\circ} \times \cos -37.82^{\circ} \times \cos 267.387^{\circ})\big)\\ &=-0.404142814063\ldots \end{align*} \begin{align*} \text{ distance} &=\frac{6400 \times \pi}{180} \times \cos^{-1}(-0.404142814063)\\ &=12715.762\ldots\\ &\approx 12716\text{ km} \end{align*}

Or in one operation with a graphic calculator,

$$d=\frac{6400 \times \pi}{180} \times \cos^{-1} \big((\sin {37.783} \times \sin {-37.82} ) + (\cos {37.783} \times \cos {-37.82} \times \cos {267.387})\big)$$ $$= 12715.762\ldots$$ $$\approx 12716\text{ km}$$

Next page - Extension Activity - The calculation