[EDIT] As Roland mentioned in comment, in a standard Python implementation, this post does not offer any solution to improve CPU performances.
In the standard Python implementation, threads do not really improve performance on CPU-bound tasks. There is a "Global Interpreter Lock" that enforces that only one thread at a time can be executing Python bytecode. This was done to keep the complexity of memory management down.
Have you tried to use different Threads for the different functions?
Let's say you separate your dataframe into columns and create multiple threads. Then you assign each thread to apply a function to a column. If you have enough processing power, you might be able to gain a lot of time:
from threading import Thread
import pandas as pd
import numpy as np
from queue import Queue
from time import time
# Those will be used afterwards
N_THREAD = 8
q = Queue()
df2 = pd.DataFrame() # The output of the script
# You create the job that each thread will do
def apply(series, func):
df2[series.name] = series.map(func)
# You define the context of the jobs
def threader():
while True:
worker = q.get()
apply(*worker)
q.task_done()
def main():
# You import your data to a pandas dataframe
df = pd.DataFrame(np.random.randn(100000,4), columns=['A', 'B', 'C', 'D'])
# You create the functions you will apply to your columns
func1 = lambda x: x<10
func2 = lambda x: x==0
func3 = lambda x: x>=0
func4 = lambda x: x<0
func_rep = [func1, func2, func3, func4]
for x in range(N_THREAD): # You create your threads
t = Thread(target=threader)
t.start()
# Now is the tricky part: You enclose the arguments that
# will be passed to the function into a tuple which you
# put into a queue. Then you start the job by "joining"
# the queue
for i, func in enumerate(func_rep):
worker = tuple([df.iloc[:,i], func])
q.put(worker)
t0 = time()
q.join()
print("Entire job took: {:.3} s.".format(time() - t0))
if __name__ == '__main__':
main()
