So if i understand you right, you have a non-localised application, which has text in English, but numbers, dates etc are automatically formatted to the CurrentCulture of the machine on which the program runs. Leading to mixed culture strings.
Obviously the best thing to do is full localise your application with resx files etc. But I assume this would be needlessly out of scope in your case.
Numbers can have quite subtle differences, you might argue that there is no problem, or indeed that it might be better to show them in the users culture. But consider datetimes:
System.Globalization.CultureInfo.CurrentCulture = new System
.Globalization.CultureInfo("fr-FR");
Console.WriteLine(DateTime.UtcNow.ToString("U"));
//mercredi 7 décembre 2016 14:20:18
DateTimes can output full month and day names in the language specified. Presumably you would want all the text to be in english? and thus you should format it, and for consistency all numbers, currencies etc in the same culture as the text.
If you are using it throughout your app, then setting it once at the start would be a good idea
PS. Obviously the only correct English is "en-GB"
The Scope of CurrentCulture is per thread. You might want to check out msdn when deciding where to set it
https://msdn.microsoft.com/en-us/library/system.globalization.cultureinfo.currentculture(v=vs.110).aspx
https://msdn.microsoft.com/en-us/library/system.globalization.cultureinfo.currentculture(v=vs.110).aspx#ThreadCulture