I have the same Python script ruing on two machines, on the same input and giving the same output. It takes about 7 minutes to run on my laptop and 56 minutes to run on a server that I uploaded it to.
Obviously, the environment is different, but the difference in speed (8x!) was a little too egregious for me to just write it off as different environments. Both machines have plenty of memory, and the server has more cores than my laptop, but also more stuff ruing on it (plus I don't think the cores matter for this operation). It's normal for me to run stuff on both my laptop and the server. I've never had a 2x difference in speed before, not to mention a 10x one.
The script is pretty sample, most of the work is a pandas.DataFrame.groupby().aggregate(lambda), but on a large (~400 MB) table.
How can I diagnose what's causing the speed difference?
Below is what I tried so far, but I know very little about this, so it may or may not be interesting.
cProfile
I tried ruing python -m cProfile in both environments to see if any particular function call was taking a long time, but it seemed to be across the board:
Laptop:
ncalls tottime percall cumtime percall filename:lineno(function)
93 56.976 0.613 372.694 4.007 {method 'get_result' of 'pandas.lib.SeriesGrouper' objects}
16338970 46.629 0.000 250.104 0.000 aggregate_sv_calls_by_sv_id.py:42(<lambda>)
18442616 33.808 0.000 56.950 0.000 {sorted}
18442645 25.395 0.000 172.113 0.000 series.py:1033(__iter__)
78056747/78056745 15.405 0.000 15.420 0.000 {isinstance}
18446903 14.235 0.000 34.129 0.000 dtypes.py:68(is_dtype)
18443264 13.515 0.000 21.058 0.000 inteals.py:3806(dtype)
18442666 13.447 0.000 30.854 0.000 common.py:2192(is_datetime_or_timedelta_dtype)
18449428 13.250 0.000 13.250 0.000 {hasattr}
18442793 13.001 0.000 19.134 0.000 inteals.py:3833(inteal_values)
...
Server:
ncalls tottime percall cumtime percall filename:lineno(function)
312253983 175.564 0.000 275.203 0.000 {isinstance}
93 153.294 1.648 3323.806 35.740 groupby.py:1885(_aggregate_series_pure_python)
16338970 151.336 0.000 749.431 0.000 aggregate_sv_calls_by_sv_id.py:42(<lambda>)
18443129 146.950 0.000 607.143 0.000 inteals.py:2482(make_block)
18442874 136.363 0.000 529.415 0.000 series.py:120(__init__)
18443241 125.725 0.000 248.772 0.000 generic.py:2674(__setattr__)
18442596 106.353 0.000 1336.629 0.000 inteals.py:3868(get_slice)
18442617 106.296 0.000 172.363 0.000 {sorted}
18442596 89.203 0.000 2105.284 0.000 series.py:689(_get_values)
18443048 84.777 0.000 91.065 0.000 base.py:506(_get_attributes_dict)
...
Since it seemed to be across the board rather than any one function I could isolate, I tried to see if I could reduce the problem to a minimum, complete, verifiable example... This was the best I got:
%timeit
Laptop:
In [5]: df = pd.DataFrame(np.random.randint(10, size=(10000, 50)))
In [6]: %timeit df.groupby(0).sum()
100 loops, best of 3: 5.54 ms per loop
In [7]: %timeit df.groupby(0).agg(lambda x: sum(x))
1 loops, best of 3: 124 ms per loop
In [8]: %timeit df.groupby(0, 1).agg(lambda x: sum(x))
1 loops, best of 3: 155 ms per loop
Server:
In [5]: df = pd.DataFrame(np.random.randint(10, size=(10000, 50)))
In [6]: %timeit df.groupby(0).sum()
100 loops, best of 3: 6.08 ms per loop
In [7]: %timeit df.groupby(0).agg(lambda x: sum(x))
1 loops, best of 3: 215 ms per loop
In [8]: %timeit df.groupby(0, 1).agg(lambda x: sum(x))
1 loops, best of 3: 327 ms per loop
So this doesn't quite capture the 8x speed difference. It only shows about a 2x speed difference.
But this size of difference (2x) I feel like I am used to chalking up to some kind of environmental factors that maybe I won't be able to figure out, like other load on the server or something like that, or different processors. If I'm willing to write off a 2x difference as different machines, does that mean that I also have to be willing to accept an 8x difference as just different machines? Or is there more interesting stuff to dig into here?