118

Apple launched its new programming language Swift at WWDC14. In the presentation, they made some performance comparisons between Objective-C and Python. The following is a picture of one of their slides, of a comparison of those three languages performing some complex object sort:

enter image description here

There was an even more incredible graph about a performance comparison using the RC4 encryption algorithm.

Obviously this is a marketing talk, and they didn't go into detail on how this was implemented in each. I leaves me wondering though:

  1. How can a new programming language be so much faster?
  2. Are the Objective-C results caused by a bad compiler or is there something less efficient in Objective-C than Swift?
  3. How would you explain a 40% performance increase? I understand that garbage collection/automated reference control might produce some additional overhead, but this much?
Tulains Córdova
  • 39,201
  • 12
  • 97
  • 154
Yellow
  • 1,273
  • 2
  • 9
  • 6
  • I don't think objective-c is a native language. It doesn't compile down to assembler as would C++. It's also very message based which isn't very performance friendly. They're also using Python as a base for comparison that should explain a lot. Swift as stated on their website claims to implement more native C code than originally implemented in Objective-C. So I think that's where they're seeing the performance changes. The above graph is likely "native C sort in Swift" virus "Objective-C sort using Objective-C". – Reactgular Jun 02 '14 at 22:08
  • 13
    @MathewFoscarini Obj-C goes to assembler, but it's got an expensive object message dispatch mechanism. For most GUI work that doesn't matter, but for sorting it matters a lot. – Donal Fellows Jun 02 '14 at 22:12
  • @DonalFellows thanks for clarifying. I haven't used obj-c. – Reactgular Jun 02 '14 at 22:13
  • 18
    The comparison to Python is the real head-scratcher here. – asthasr Jun 02 '14 at 22:20
  • 10
    @syrion marketing, and the language appears to borrow from python's syntax (much like golang). They are trying to say "hey, python developers, you can write something not too foreign on the mac and have it be SO much faster, even faster than Objective C that you never really got the hang of" –  Jun 02 '14 at 22:47
  • 4
    @MichaelT I get that, but it's still odd. Anyone who knows anything about languages will realize that Python, as an interpreted language, simply isn't going to be in the same ballpark as Objective-C or other compiled languages (for most tasks). Using it as a benchmark is odd. – asthasr Jun 02 '14 at 22:50
  • 1
    *Caveat Emptor!* It all comes down to machine language. There probably are good ideas in *Swift*, but charts like that are meaningless. – Mike Dunlavey Jun 02 '14 at 22:50
  • @MichaelT, actually Swift is much closer to JavaScript than it is to Python.... – miraculixx Jun 03 '14 at 21:39
  • @miraculixx looking at it more closely, yep. Though they're still trying to appeal to a certain segment of the coding population. With Javascript one can already build html5 hybrid apps... this says "hey python people, you can use this to build apps for iOS and don't have to write something that looks like C". I think its very much a marketing thing trying to get another set of programmers into the iOS and Mac OS fold. –  Jun 03 '14 at 21:42
  • @MichaelT, ok I get that, and that's ok. However, the average Python developer will frown upon blocks with curly braces, and Swift has plenty :-) except these guys, of course: http://www.pythonb.org/ – miraculixx Jun 03 '14 at 21:54
  • 7
    They probably mean the time it takes to write the code... – Lukas Eder Jun 04 '14 at 13:38
  • 4
    I think this chart says a lot more about how slow Objective-C is than how fast Swift is. –  Jun 05 '14 at 01:02
  • 2
    It does not seem to be a coincidence that they decided to benchmark "complex objects". If you are sorting integers, Swift is 10 times slower than Python: http://stackoverflow.com/questions/24101718/swift-performance-sorting-arrays – Jukka Suomela Jun 08 '14 at 09:44
  • I would pick dynamic dispatch as the killer. So exactly how does Swift do it, and there you have your answer. – david.pfx Jun 23 '14 at 00:00
  • @syrion I thought about the python thing too. I think the point is to avoid comparison with a fast language. Python is one of the faster scripting languages I've seen around, but last time I checked it was still 10x slower than Java and 20x slower than C for most stuff. If they were to stick either of those in there Swift wouldn't have looked quite as swift. – Bill K Nov 14 '14 at 02:22

5 Answers5

72

Being 3.9 times faster than python, the language that consistently loses most benchmarks by considerable margin (ok, it's on par with Perl, Ruby and PHP; but it loses to anything statically typed), is nothing one should be boasting about.

The benchmarks game shows C++ programs that are more than order of magnitude faster than python programs in most cases. It is not much better when comparing with Java, C# (on Mono), OCaml, Haskell and even Clojure, which is not statically typed.

So the real question is how is Objective-C only 2.8 times faster than python. Apparently they carefully chose benchmark where the slow fully dynamic dispatch of ObjC hurts a lot. Any statically typed language should be able to do better. In complex object sort there are many method calls for comparing the objects and the actual comparison was probably not very complex itself. So if Swift takes at least some advantage of the type information, it can easily do better on the method calls and there is not enough other operations that ObjC could be better in.

Of course, as the benchmarks game clearly demonstrates, the relative performance on different tasks varies wildly, so one benchmark is not really representative. If they had benchmark where it had bigger advantage, they would have shown us that one instead, so on other tasks it probably isn't better or not so much.

igouy
  • 574
  • 3
  • 6
Jan Hudec
  • 18,250
  • 1
  • 39
  • 62
  • 13
    I don't quite understand the point of this answer. Are you answering "how is swift faster" by saying "the benchmarks are flawed"? Is that the point you're making? I don't see how that answers what was being asked. – Bryan Oakley Jun 03 '14 at 11:57
  • 15
    @BryanOakley: I don't think the benchmarks are flawed, but the possibility that they picked the one benchmark where Swift was faster certainly has to be considered. – Jan Hudec Jun 03 '14 at 12:18
  • 1
    You make good points about comparison Objective-C with Python, which doesn't seem fair at all in my mind. I wonder if you have any explanation for the difference with Swift though, which are both compiled and typed languages. – Yellow Jun 03 '14 at 12:30
  • 1
    @Yellow: ObjC contains the statically typed C sublanguage, but the objective layer has fully dynamic dispatch. So it's not difficult to do better in method calls and consequently in benchmark that involves a lot of method calls and not much other work. – Jan Hudec Jun 03 '14 at 12:38
  • 24
    It's possible that the answer to "How is Swift faster?" might be "It's not, actually", @BryanOakley; that is the gist that I get from Jan's answer. "Lies, damn lies, and statistics", after all. – jscs Jun 03 '14 at 19:51
  • >>"Lies, damn lies, and statistics", after all.<< "After all, facts are facts, and although we may quote one to another with a chuckle the words of the Wise Statesman, 'Lies--damned lies--and statistics,' still there are some easy figures the simplest must understand, and the astutest cannot wriggle out of." Leonard Henry Courtney, 1895 – igouy Jun 03 '14 at 20:38
  • Give Python some credit: "loses most benchmarks", well it depends on *what* benchmarks. It generally outperforms Java, C and C++ when it comes to programmer productivity, e.g. as shown by this study (PDF) http://www.connellybarnes.com/documents/language_productivity.pdf. – miraculixx Jun 03 '14 at 21:35
  • 1
    @igouy "facts are facts", yes. In this case we don't know the facts, not the least bit. All we know is a claim with no context, rendering it utterly useless and potentially deceptive. Specifically, we don't know how the sort was implemented in any of these languages, if some optimized library was used in the case of Swift, nor what flavor of Python they used (CPython, Cython, ...). A fair benchmark would state all of that, plus probably leave Python out of it in the first place as the language is not a competitor to Objective C or Swift. If anything they should have compared to JavaScript. – miraculixx Jun 03 '14 at 21:49
  • 4
    A while back we benchmarked Codename One running on iOS and our Java implementation was much faster than Objective-C http://www.codenameone.com/blog/codename-one-benchmarked-with-amazing-results Jan is correct, dynamic dispatch is really slow, if they improved them even a little bit then some benchmarks will show a huge improvement. If they improve ARC even by a fraction (thanks to better code analysis) they can get rid of a ridiculous amount of complexity. The more restricted the language the more the compiler can do to optimize (see Java) and Swift adds restrictions. – Shai Almog Jun 04 '14 at 05:34
  • @miraculixx "In this case we don't know the facts" -- I agree with your specific criticisms which so unlike that vague dismissal "Lies, damn lies, and statistics". – igouy Jun 05 '14 at 15:52
  • 8
    Jan's answer is a perfect answer to the Q1 and probably Q2. When I saw the benchmarks on a marketing event as the keynote, I thought: "Wow, only 1,3x in the selected best-case. What will we the average result? 0.3x?" – Amin Negm-Awad Jun 09 '14 at 08:30
  • According to this benchmark "GoodManWEN Programming-Language-Benchmarks-Visualization" Swift is 10 times faster than Python. But still very slow compared to C++ and Rust. – Tom Feb 21 '23 at 16:25
62

First, (IMO) comparing with Python is nearly meaningless. Only comparison with Objective-C is meaningful.

  • How can a new programming language be so much faster?

Objective-C is a slow language. (Only C part is fast, but that's because it's C) It has never been extremely fast. It was just fast enough for their (Apple's) purpose, and faster then their older versions. And it was slow because...

  • Do the Objective-C results from a bad compiler or is there something less efficient in Objective-C than Swift?

Objective-C guaranteed every method to be dynamically dispatched. No static dispatch at all. That made it impossible to optimize an Objective-C program further. Well, maybe JIT technology can be some help, but AFAIK, Apple really hate unpredictable performance characteristics and object lifetime. I don't think they had adopted any JIT stuff. Swift doesn't have such dynamic dispatch guarantee unless you put some special attribute for Objective-C compatibility.

  • How would you explain a 40% performance increase? I understand that garbage collection/automated reference control might produce some additional overhead, but this much?

GC or RC doesn't matter here. Swift is also employing RC primarily. No GC is there, and also will not unless there's some huge architectural leap on GC technology. (IMO, it's forever) I believe Swift has a lot more room for static optimization. Especially low level encryption algorithms, as they usually rely on huge numeric calculations, and this is a huge win for statically dispatch languages.

Actually I was surprised because 40% seems too small. I expected far more. Anyway, this is the initial release, and I think optimization was not the primary concern. Swift is not even feature-complete! They will make it better.

Update

Some keep bugging me to argue that the GC technology is superior. Though stuff below can be arguable, and just my very biased opinion, but I think I have to say to avoid this unnecessary argument.

I know what conservative/tracing/generational/incremental/parallel/realtime GCs are and how they are different. I think most of readers also already know that. I also agree that GC is very nice in some field, and also shows high throughput in some cases.

Anyway, I suspect the claim of GC throughput is always better than RC. Most of overhead of RC comes from ref-counting operation and locking to protect ref-count number variable. And RC implementation usually provide a way to avoid counting operations. In Objective-C, there's __unsafe_unretained and in Swift, (though it's still somewhat unclear to me) unowned stuffs. If the ref-counting operation cost is not acceptable, you can try to opt-out them selectively by using the mechanics. Theoretically, we can simulate almost unique-ownership scenario by using non-retaining references very aggressively to avoid RC overhead. Also I expect the compiler can eliminate some obvious unnecessary RC operations automatically.

Unlike RC system, AFAIK, partial opt-out of reference-types is not an option on GC system.

I know there're many released graphics and games which are using GC based system, and also know most of them are suffering by lack of determinism. Not only for performance characteristic, but also object lifetime management. Unity is mostly written in C++, but the tiny C# part causes all the weird performance issues. HTML hybrid apps and still suffering by unpredictable spikes on any system. Used widely doesn't mean that's superior. It just means that's easy and popular to people who don't have many options.

Update 2

Again to avoid unnecessary argument or discussion, I add some more details.

@Asik provided an interesting opinion about GC spikes. That's we can regard value-type-everywhere approach as a way to opt-out GC stuff. This is pretty attractive, and even doable on some systems (purely functional approach for example). I agree that this is nice in theory. But in practice it has several issues. The biggest problem is partial application of this trick does not provide true spike-free characteristics.

Because latency issue is always all or nothing problem. If you have one frame spike for 10 seconds (= 600frames), then the whole system is obviously failing. This is not about how better or worse. It's just pass or fail. (or less then 0.0001%) Then where is the source of GC spike? That's bad distribution of GC load. And that's because the GC is fundamentally indeterministic. If you make any garbage, then it will activate the GC, and spike will happen eventually. Of course, in the ideal world where GC load will be always ideal, this won't happen, but I am living in real world rather than imaginary ideal world.

Then if you want to avoid spike, you have to remove all the ref-types from the whole system. But it's hard, insane, and even impossible due to unremovable part such as .NET core system and library. Just using non-GC system is far easier.

Unlike GC, RC is fundamentally deterministic, and you don't have to use this insane optimization (purely-value-type-only) just only to avoid spike. What you have to do is tracking down and optimizing the part which causes the spike. In RC systems, spike is local algorithm issue, but in GC systems, spikes are always global system issue.

I think my answer is gone too much off-topic, and mostly just repetition of existing discussions. If you really want to claim some superiority/inferiority/alternative or anything else of GC/RC stuffs, there're plenty of existing discussions in this site and StackOverflow, and you can continue to fight at there.

Eonil
  • 1,719
  • 1
  • 13
  • 28
  • 4
    Garbage collection, especially generational, is usually _significantly faster_ than reference counting. – Jan Hudec Jun 05 '14 at 16:33
  • 10
    @JanHudec Your *significantly faster* is simply meaningless in realtime graphics field. That's why I mention *huge leap* is required on GC. Generational GC is not even close to be spike-free both in theoretically, and practically. – Eonil Jun 05 '14 at 21:37
  • 6
    _Faster_ and _spike free_ are completely orthogonal categories. Generational garbage collectors are _faster_. They are _not_ spike free. – Jan Hudec Jun 05 '14 at 21:46
  • 4
    What you're talking is *throughput*. *Faster* has been always a vague term, and can mean anything by the context. If you want to argue about meaning of terms, you should use more precise term rather than *faster* especially considering current context - realtime graphics. – Eonil Jun 05 '14 at 22:32
  • 1
    Whatever. While I am sure there are areas where the spikes are a problem, note that there are now many games written in Java, various incarnations of .NET and some other garbage collected languages, including many fast paced games, and nobody seems to complain about seeing any spikes. – Jan Hudec Jun 06 '14 at 06:28
  • 1
    "partial opt-out is not an option on GC system" - .NET Value Types are not tracked by GC and can be extensively used where low latency is primordial. Also, of course unmanaged allocations + refcounting can be implemented in a language that has GC (just like GC can be implemented in a language where the runtime doesn't offer it). – Asik Jun 13 '14 at 17:49
  • 1
    @Asik (1) GC/RC is invented only to serve ref-types - which have an identity. Then value-type doesn't seem to be in focus of this discussion. In addition, value-types can be equally available on any system, so it doesn't provide comparable differences. (2) The idea of allocating GC classes in unmanaged memory is interesting. Is this possible on .NET? I want you to provide some more details. I think I have to change my opinion if this is possible. (3) Latency issue is still remains because ref-types couldn't be removed completely. – Eonil Jun 14 '14 at 01:29
  • You could overload equality on a value type to compare by some id to provide identity semantics and pass them around by reference - the point is that .NET (unlike Java for instance) supports user-defined types not tracked by GC and this is a way to avoid latency spikes. The technique is widely used in XNA, Unity games, etc. Regarding refcounting, I'm just saying you can implement an unmanaged resource pool with .NET objects responsible for their allocation and recycling with reference counts, it's just more verbose than in a language natively doing ARC. – Asik Jun 14 '14 at 03:56
  • @Asik OK I see your point. I agree that you can use value-type approach to avoiding GC, but I don't think that's an *opt-out*. It's just avoiding. Well, this seems to be a subtle difference of wording choice. Anyway I'll update my answer to reflect that. And I will regard unmanaged memory allocation stuff is something only for alien code (= external resource) unless you provide some proper details. – Eonil Jun 14 '14 at 05:57
  • 12
    @JanHudec: On mobile devices, or any device with constrained resources, [GC is *not* significantly faster](http://sealedabstract.com/rants/why-mobile-web-apps-are-slow/) and in fact is a major part of the problem. – Mason Wheeler Jun 14 '14 at 13:18
  • @Eonil There are some truly spike-free garbage collectors. The Azul Zing concurrent compacting collector does not have any spikes beyond a few milliseconds. Some collectors are designed for hard realtime applications and have similar guarantees. – Demi Oct 22 '14 at 04:19
  • Also Lua (a garbage collected language) is widely used in games. The key is that the GC must be incremental, which usually means non-moving. The Lua GC is incremental, C#'s is not. – Demi Sep 30 '15 at 01:38
  • 1
    The answer is correct, but the part regarding GC is but a misleading flame-war bait. TL; DR: GC is not inferior or superior, because avoiding memory (re)allocations completely is the only right path to performance. Both GC and memory allocator are expensive shared resources; limiting their use is the only right way to write high-performant code. If you reuse your allocations, the difference between RC, plain malloc and GC-based systems becomes irrelevant. – user1643723 Aug 06 '17 at 06:57
5

Objective-C dynamically dispatches every method call.

Hypothesis: The benchmark uses static typing to let the Swift compiler hoist the compare method lookup out of the sort loop. This requires a narrow type restriction that only allows Complex objects in the array, not subclasses of Complex.

(In Objective-C you could hoist the method lookup manually if you really want to, by calling on the language runtime support to look up the method pointer. You'd better be sure that all instances in the array are of the very same class.)

Hypothesis: Swift optimizes reference-counting calls out of the loop.

Hypothesis: The Swift benchmark uses a Complex struct in place of an Objective-C object, so sort comparisons don't need dynamic method dispatches (since it can't be subclassed) or reference-counting work (since it's a value type).

(In Objective-C you can fall back to C/C++ for performance as long as it doesn't involve Objective-C objects, e.g. sort a C array of structs.)

Jerry101
  • 5,367
  • 1
  • 15
  • 19
3

Honestly, unless they release the source to the tests they are using I wouldn't trust anything Apple has to say on the subject. Remember, this is the company which switched from PPC to Intel based on power concerns when 6 months earlier they were saying that Intel sucked and actually torched the Intel bunny in a commercial. I would like to see hard irrefutable proof that Swift is faster than ObjC in more categories than just sorting.

Additionally you have to question any stats which are released at WWDC since they have the smell of marketing all over them.

All of that being said I haven't run any tests between swift and ObjC myself, but from what I know swift has it's own LLVM IR extensions and it is possible that more optimization is being done at compile time than in ObjC.

Full Disclosure: I am writing an open source Swift compiler located at https://ind.ie/phoenix/

If anyone would like to help make sure that Swift isn't just available on Apple hardware, let me know and I would be glad to include you.

greg.casamento
  • 201
  • 1
  • 4
0

I've struggled myself through the Swift tutorial, and it seems to me that Swift is more down to earth (makes me think of Visual Basic) with less 'object-ification' than Objective-C. Had they taken into account C or C++, I presume the latter would have won, since they are even more compile-time only.

In this case, I presume Objective-C is the victim of it's object oriented pureness (and overhead).

Painted Black
  • 153
  • 1
  • 13
    Doing a "complex object sort" benchmark would be a bit difficult in a language such as C that doesn't have a native implementation of objects. Given that the audience for this talk was likely 100% Objective-C programmers, comparing to a language like C++ doesn't make much sense either. The point of swift isn't "hey, this is the greatest language ever!" but rather "hey, this is faster than the language you're using now for OSX/iOS development". – Bryan Oakley Jun 03 '14 at 12:00
  • 10
    C has a perfectly fine `qsort` which will allow complex object sorting; it just uses a callback which understands the objects at hand. I suspect C++ is missing because `std::sort` would embarrass Swift. (Since it's a template, a C++ compiler can heavily optimize it, up to loop unrolling.) – MSalters Jun 04 '14 at 10:46
  • @MSalters: I fully agree with you. Both C and C++ have capabilities to outrun Swift. Would it be possible to get the executed test. I'm more than willing to execute the same benchmark with Swift, Objective-C, C++ and C. – Painted Black Jun 04 '14 at 21:07
  • @BryanOakley also, "This language requires less square brackets!" – Nick Bedford Jun 20 '14 at 02:57
  • 1
    This answer doesn't hold water at all and is horribly misleading. OO is not really slow, in fact the fastest systems you'll find around are going to be C++, Java and C# and the style (heavily OO or not) of programming will have very little to do with the resultant speeds unless you have really bad code. – Bill K Nov 14 '14 at 02:43