10 Recommended Python libraries in 2016
Pytest is a full-featured Python testing framework that helps you test your program.
At first let us install the tool by insert the following command in CMD:
Note that if you work with Spyder No installation is required.
pip install -U pytest
Now let us check an arbitrary function by creating test1.py file containg the next functions:
def Plus(a,b): return a + b def test_answer(): assert Plus(3,2) == 5
The function assert running a test on your function.
In order to Receive the test results. We use the following line on CMD:
Those two rows are identical:
pytest test1.py python -m pytest test1.py
In the CMD that looks like this:
pytest C:\Users\USER\Desktop\test1.py ============================= test session starts ============================= platform win32 -- Python 2.7.12, pytest-3.0.0, py-1.4.31, pluggy-0.3.1 rootdir: C:\Users\USER, inifile: collected 1 items Desktop\test1.py . ========================== 1 passed in 0.02 seconds ===========================
For a verbal display let's insert the next lines at CMD:
python -m pytest -v test1.py py.test -v test1.py
============================= test session starts ============================= platform win32 -- Python 2.7.12, pytest-3.0.0, py-1.4.31, pluggy-0.3.1 -- C:\Users\USER\Anaconda2\python.exe cachedir: .cache rootdir: C:\Users\USER, inifile: collected 1 items Desktop/test1.py::test_answer PASSED ========================== 1 passed in 0.01 seconds ===========================
The test has been passed, namely, the function works correctly.
Now let’s run a test for a failed result:
def Plus(a,b): return a + b def test_answer(): assert Plus(3,2) == 4
At CMD Let's Input:
pytest test1.py python -m pytest test1.py
And the output will be:
_________________________________ test_fun ___________________________________ def test_fun(): > assert Plus(3,2) == 4 E assert 5 == 4 E + where 5 = Plus(3, 2) Desktop\test1.py:12: AssertionError ========================== 1 failed in 0.05 seconds ===========================
Note that assert function saves the need for understanding of the components of a particular function, and gives you the value it expects to receive.
The same a test can be run more simply in Spyder by import pytest and call the testing function:
In : import pytest
In case the test succeed the function dose not returne anything, otherwise will be returned:
In : test_fun() Traceback (most recent call last): File "<ipython-input-32-97a1938dfe72>", line 1, in <module> test_fun() File "<ipython-input-30-60cba3ee17f2>", line 5, in test_fun assert Plus(3,2) == 4 AssertionError
Running multiple tests with a class:
class TestClass: def test_one(self): x = "Discover" assert 'D' in x def test_two(self): y = "SDK" assert abs(-4) == 3
Running the file at the command window will return:
================================== FAILURES =============================== _____________________________ TestClass.test_two ______________________________ self = <test1.TestClass instance at 0x0000000003D19108> def test_two(self): y = "SDK" > assert abs(-4) == 3 E assert 4 == 3 E + where 4 = abs(-4) Desktop\test1.py:14: AssertionError 1 failed, 1 passed in 0.05 seconds
We can see that the first function has succeeded, while the second function had failed. assertion, Gives us an indication of the reason for the failure.
assert that exception is raised:
In order to assert that some code raises an exception you can use raises:
import pytest def test(): with pytest.raises(BaseException): raise BaseException(1)
And the CMD will be looked like:
============================= test session starts ============================= platform win32 -- Python 2.7.12, pytest-3.0.0, py-1.4.31, pluggy-0.3.1 rootdir: C:\Users\USER, inifile: collected 1 items Desktop\test1.py. ========================== 1 passed in 0.03 seconds ===========================
Pyprind(Python Progress Indicator)
Pyprind provides a progress bar and a percentage indicator object.
Allows you monitor the progression of various activities such as loops and various calculations.
Very useful action is tracking the progress of large data sets and providing an effective runtiem estimate about the progression.
For installation let us open the CMD and type:
In : pip install pyprind Or In : easy_install pyprind In : import pyprind
In : import time n = 50 sign = pyprind.ProgBar(n, bar_char='*') for i in range(n): time.sleep(0.1) # your computation here sign.update()
We'll get the next Progress Bar:
0% 100% [******************************] | ETA: 00:00:00 Total time elapsed: 00:00:05 perc = pyprind.ProgPercent(n) for i in range(n): time.sleep(timesleep) # your computation here perc.update()
In :percent = pyprind.ProgPercent(n) for i in range(n): time.sleep(0.1) percent.update() [100 %] Time elapsed: 00:00:05 | ETA: 00:00:00 Total time elapsed: 00:00:05
You can get more specific information on the tracking of the loop by adding a command print(sign):
sign = pyprind.ProgBar(n) for i in range(n): time.sleep(0.1) sign.update() print(sign) 0% 100% [##############################] | ETA: 00:00:00 Total time elapsed: 00:00:05 Title: Started: 08/24/2016 10:35:28 Finished: 08/24/2016 10:35:33 Total time elapsed: 00:00:05
We Can also monitored the memory and CPU by adding the following command:
sign = pyprind.ProgBar(n, monitor=True) 0% 100% [##############################] | ETA: 00:00:00 Total time elapsed: 00:00:10 Title: Started: 08/24/2016 10:42:28 Finished: 08/24/2016 10:42:38 Total time elapsed: 00:00:10 CPU %: 0.30 Memory %: 1.78
Setting Title, Change the bar width, Determining the scale icon.
We can easily set a title for the report, by adding title='tiltle' in pyprind.ProgBar. for changing the bar width add width= . for choosing the scale icon add bar_char='sign'. in pyprind.ProgBar.
sign = pyprind.ProgBar(n, title='prog1',width=5,bar_char='█' ) for i in range(n): time.sleep(0.2) sign.update() prog1 0% 100% [█████] | ETA: 00:00:00 Total time elapsed: 00:00:10
Stopping the process
You can perform a stop at selected index:
sign = pyprind.ProgBar(n) for i in range(n): time.sleep(0.1) if i == 25: sign.stop() sign.update() 0% 100% [##############################] | ETA: 00:00:00 Total time elapsed: 00:00:02 Total time elapsed: 00:00:05
Update interval time
Update process at selected interval by adding update_interval:
n = 50 bar = pyprind.ProgBar(n, update_interval=2 , width=5) for i in range(n): time.sleep(0.2) bar.update() 0% 100% [#####] | ETA: 00:00:00 Total time elapsed: 00:00:10
tqdm let you add A progress bar for examine your loops running
for installation, insert the next command into CMD:
pip install tqdm
Now let us go beck to python and ipmort tqdm library:
In : from tqdm import tqdm
At tqdm unlike pyprind we will use with sentence
with tqdm(total=100) as barval: for i in range(5): barval.update(20) 100%|██████████| 100/100 [00:00<?, ?it/s]
You can give up the with sentence by adding .close ():
barval = tqdm(total=100) for i in range(100): barval.update(1) barval.close() 100%|██████████| 100/100 [00:00<?, ?it/s]
Unlike tqdm, here there is no need to use the Print command to see the progress data.
nested loops progress
from tqdm import trange
trange(i) is a special optimised instance of tqdm(range(i)):
from tqdm import trange from time import sleep for i in trange(5, desc='first'): for j in trange(1, desc='scond', leave=False): for k in trange(3, desc='third'): sleep(0.1)
Add a Comment
You can print a title for each loop by using tqdm.write
val = trange(5) for i in val: sleep(0.2) if not (i == 2): tqdm.write("Loop %i" % i) Loop 0 | 0/5 [00:00<?, ?it/s] Loop 1 | 1/5 [00:00<00:00, 4.26it/s] Loop 3█████ | 3/5 [00:00<00:00, 4.55it/s] Loop 4 ███ | 4/5 [00:00<00:00, 4.41it/s] 100%|██████████| 5/5 [00:01<00:00, 4.44it/s]
unittest2 — Unit testing framework
Unittest2 module provides a rich set of tools for running tests.
Test has two parts, code to manage “fixtures”, and the test itself. Individual tests are created by subclassing unittest.TestCase and overriding or adding appropriate methods.
unittest2 is a backport of the features that are added to the unittest framework in Python 2.7 and forward versions.
Let's takt a look at some innovations which are added to Unittest2:
- addCleanups allows better resource management
- Assert methods renewals such as better defaults for lists comparing , sets, dicts unicode strings and more. In addition able to specify new default methods for comparing specific types.
- assertRaises as context manager, with access to the exception afterwards
- test discovery and new command line options.
- class and module level fixtures: setUpClass, tearDownClass, setUpModule, tearDownModule
- Allows to skip specific tests and even classics, and can Indicates on expected failures.
- new delta keyword argument to assertAlmostEqual for non-numeric objects comparisons .
pip install unittest2
Let's take a look at a simple example:
Note that this example ran directly from the python shell.
import unittest2 class SimplisticTest(unittest.TestCase): def test(self): self.assertEqual('sdk'.upper(), 'SDK') if __name__ == '__main__': unittest.main() ---------------------------------------------------------------------- Ran 1 test in 0.000s
Unittest can be used from the CMD by the next command:
Note that you need to redirect by cd cammand in CMD to the folder where the file is:
python -m unittest test_module1 test_module2 python -m unittest test_module.TestClass python -m unittest test_module.TestClass.test_method
You can run more detailed tests (verbosity) by:
python -m unittest -v test_module
The following command in CMD will show us a list of all command-line possibilities:
python -m unittest -h
To allow proper operation of the Test Discovery
Unittest allows skipping specific test and whole classes of tests if necessary.
Kivy is an open source Python library enabling easy and fast application development. Using user interfaces technologies such as multi-touch apps.
Kivy is based on OpenGL ES 2. In addition it supports diverse input devices and has a wide gadgets library.
Open the CMD window and type the next lines:
python -m pip install --upgrade pip wheel setuptools python -m pip install docutils pygments pypiwin32 kivy.deps.sdl2 kivy.deps.glew
Note that you can skip gstreamer instalation if not necessary(90MB)
python -m pip install kivy.deps.gstreamer --extra-index-url https://kivy.org/downloads/packages/simple/ python -m pip install kivy
Now there should be no problem to importe kivi.
Let’s take a look at a simple examle:
from kivy.app import App from kivy.uix.button import Button from kivy.app import App from kivy.uix.button import Button class TestApp(App): def build(self): return Button(text='Discover SDK') TestApp().run()
Now the next separate window should be opened:
A range of more examples you cam find here.
Keras is a Deep Learning library for Theano and TensorFlow
In addition it supports convolutional networks and recurrent networks Separately and together.
Keras is adjusted for Python 2.7-3.5.
Before start Keras instalation be sure that you have Theano or TensorFlow.
For example Theano instalation will be done by the next command(CMD):
pip install Theano
Note that Keras uses Theano as its tensor manipulation library By default. you can say that Keras is a Theano's wrapper designed to make it more accessible for useres.
Keras uses the following libraries:
- numpy, scipy, pyyaml, HDF5 and h5py and cuDNN if you use CNNs.
pip install numpy scipy pip install scikit-learn pip install pillow pip install h5py
Now let's install Kras:
pip install keras
Now let's check if Keras has been installed successfully by import it to the Python shell:
from keras.models import Sequential from keras.layers import Dense, Activation model = Sequential() model = Sequential([ Dense(output_dim=32, input_dim=784), Activation('relu'), Dense(10), Activation('softmax'), ])
we also can add layers by
model.add(Dense(output_dim=64, input_dim=784)) model.add(Activation('relu'))
When your model si set and before you move for the training model , you should formate the learning process. This can be done by
compile class. whcih receives three arguments: an optimizer, a loss function and a list of metrics
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
For training yout model, Let's use the
model.fit(train1, train2, nb_epoch=5, batch_size=32) Epoch 1/5 1000/1000 [==============================] - 0s - loss: 0.7270 - acc: 0.4960 Epoch 2/5 1000/1000 [==============================] - 0s - loss: 0.7117 - acc: 0.5180 Epoch 3/5 1000/1000 [==============================] - 0s - loss: 0.7064 - acc: 0.5240 Epoch 4/5 1000/1000 [==============================] - 0s - loss: 0.6977 - acc: 0.5300 Epoch 5/5 1000/1000 [==============================] - 0s - loss: 0.6843 - acc: 0.5730 Out: <keras.callbacks.History at 0xf563a58>
For Evaluatung your performance:
loss_and_metrics = model.evaluate(train1, train2, batch_size=32) 960/1000 [===========================>..] - ETA: 0s
TFlearn is a modular deep learning library run on top of Tensorflow. It wrappes the Tensorflow in order to make it more user friendly and to accelerate the
experimentations, by provinding a higher-level API.
TFlearn also supports most of recent deep learning models, such as Convolutions, LSTM, BiRNN, BatchNorm
pip install tflearn
The following example taken from the official website of TFlearn and demonstrates the effectiveness of its API:
with tf.name_scope('conv1'): W = tf.Variable(tf.random_normal([5, 5, 1, 32]), dtype=tf.float32, name='Weights') b = tf.Variable(tf.random_normal(), dtype=tf.float32, name='biases') x = tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME') x = tf.add_bias(W, b) x = tf.nn.relu(x)
The same operation is written more efficiently with TFlearn:
tflearn.conv_2d(x, 32, 5, activation='relu', name='conv1')
Nose2 is the update for Nose the unit test framework and based on unittest2.
several effective changes had made since the firs generation nose, Including:
- A significant improvement in plugin api
- More accessible and more user-friendly
- simplifying internal Processes.
- compatible with Python 2.6-3.4 based on the same codebase, without translation.
However, some of nose features are no supprted by nose2.. you will able to find them here.
pip install nose2
In order to run the test, the code name have to starts with the word test. any other combination of the names will not work. for eample:
test1.py ------> will wotk example1.py ------> will not work
We want to examine the following functions:
We would expect to ger test failed for the second function.
def multiply(a,b): return a*b def test_num_mul(): assert multiply(5,6) == 30 def test_str_mul(): assert multiply('g',5) == 'ggg'
Now we can run the test by the following command at CMD:
Note that it is necessary to redirect with cd to the file's folder.
And the output much like uinitest, will be:
.F ====================================================================== FAIL: test1.FunctionTestCase (test_str_mul) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\USER\Desktop\test1.py", line 15, in test_str_mul assert multiply('g',5) == 'ggg' AssertionError ---------------------------------------------------------------------- Ran 2 tests in 0.000s FAILED (failures=1)
we can get help for nose2 plugins by:
usage: nose2 [-s START_DIR] [-t TOP_LEVEL_DIRECTORY] [--config [CONFIG]] [--no-user-config] [--no-plugins] [--plugin PLUGINS] [--exclude-plugin EXCLUDE_PLUGINS] [--verbose] [--quiet] [--log-level LOG_LEVEL] [-B] [--log-capture] [-D] [--coverage PATH] [--coverage-report TYPE] [--coverage-config FILE] [-C] [-F] [-h] [testNames [testNames ...]] positional arguments: testNames optional arguments: -s START_DIR, --start-dir START_DIR Directory to start discovery ('.' default) -t TOP_LEVEL_DIRECTORY, --top-level-directory TOP_LEVEL_DIRECTORY, --project-directory TOP_LEVEL_DIRECTORY Top level directory of project (defaults to start dir) --config [CONFIG], -c [CONFIG] Config files to load, if they exist. ('unittest.cfg' and 'nose2.cfg' in start directory default) --no-user-config Do not load user config files --no-plugins Do not load any plugins. Warning: nose2 does not do anything if no plugins are loaded --plugin PLUGINS Load this plugin module. --exclude-plugin EXCLUDE_PLUGINS Do not load this plugin module --verbose, -v print test case names and statuses --quiet --log-level LOG_LEVEL Set logging level for message logged to console. -h, --help Show this help message and exit plugin arguments: Command-line arguments added by plugins: -B, --output-buffer Enable output buffer --log-capture Enable log capture -D, --debugger Enter pdb on test fail or error --coverage PATH Measure coverage for filesystem path (multi-allowed) --coverage-report TYPE Generate selected reports, available types: term, term-missing, annotate, html, xml (multi-allowed) --coverage-config FILE Config file for coverage, default: .coveragerc -C, --with-coverage Turn on coverage reporting -F, --fail-fast Stop the test run after the first error or failure
TensorFlow is an open source software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. The flexible architecture allows you to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API.
The TensorFlow Python API supports Python 2.7 and Python 3.3 and forward.
The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, plots an so on. It Has become a very popular tool for scientific processes. Allows you to run your work and get results immediately in real time.
Although i put it here under the association to Python. It has support for many additional languages. In fact over 40.
including: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and more.
There are several ways to install Jupiter, one of them and the convenient in my opinion , is by downloading and installing Anaconda. Jupiter can be run directly from Anaconda.