METADATA
4.62 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
Metadata-Version: 2.1
Name: cloudpickle
Version: 1.3.0
Summary: Extended pickling support for Python objects
Home-page: https://github.com/cloudpipe/cloudpickle
Author: Cloudpipe
Author-email: cloudpipe@googlegroups.com
License: BSD 3-Clause License
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: POSIX
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: System :: Distributed Computing
Description-Content-Type: text/markdown
# cloudpickle
[](https://github.com/cloudpipe/cloudpickle/actions)
[](https://codecov.io/github/cloudpipe/cloudpickle?branch=master)
`cloudpickle` makes it possible to serialize Python constructs not supported
by the default `pickle` module from the Python standard library.
`cloudpickle` is especially useful for **cluster computing** where Python
code is shipped over the network to execute on remote hosts, possibly close
to the data.
Among other things, `cloudpickle` supports pickling for **lambda functions**
along with **functions and classes defined interactively** in the
`__main__` module (for instance in a script, a shell or a Jupyter notebook).
Cloudpickle can only be used to send objects between the **exact same version
of Python**.
Using `cloudpickle` for **long-term object storage is not supported and
strongly discouraged.**
**Security notice**: one should **only load pickle data from trusted sources** as
otherwise `pickle.load` can lead to arbitrary code execution resulting in a critical
security vulnerability.
Installation
------------
The latest release of `cloudpickle` is available from
[pypi](https://pypi.python.org/pypi/cloudpickle):
pip install cloudpickle
Examples
--------
Pickling a lambda expression:
```python
>>> import cloudpickle
>>> squared = lambda x: x ** 2
>>> pickled_lambda = cloudpickle.dumps(squared)
>>> import pickle
>>> new_squared = pickle.loads(pickled_lambda)
>>> new_squared(2)
4
```
Pickling a function interactively defined in a Python shell session
(in the `__main__` module):
```python
>>> CONSTANT = 42
>>> def my_function(data):
... return data + CONSTANT
...
>>> pickled_function = cloudpickle.dumps(my_function)
>>> pickle.loads(pickled_function)(43)
85
```
Running the tests
-----------------
- With `tox`, to test run the tests for all the supported versions of
Python and PyPy:
pip install tox
tox
or alternatively for a specific environment:
tox -e py37
- With `py.test` to only run the tests for your current version of
Python:
pip install -r dev-requirements.txt
PYTHONPATH='.:tests' py.test
Note about function Annotations
-------------------------------
Note that because of design issues `Python`'s `typing` module, `cloudpickle`
supports pickling type annotations of dynamic functions for `Python` 3.7 and
later. On `Python` 3.4, 3.5 and 3.6, those type annotations will be dropped
silently during pickling (example below):
```python
>>> import typing
>>> import cloudpickle
>>> def f(x: typing.Union[list, int]):
... return x
>>> f
<function __main__.f(x:Union[list, int])>
>>> cloudpickle.loads(cloudpickle.dumps(f)) # drops f's annotations
<function __main__.f(x)>
```
History
-------
`cloudpickle` was initially developed by [picloud.com](http://web.archive.org/web/20140721022102/http://blog.picloud.com/2013/11/17/picloud-has-joined-dropbox/) and shipped as part of
the client SDK.
A copy of `cloudpickle.py` was included as part of PySpark, the Python
interface to [Apache Spark](https://spark.apache.org/). Davies Liu, Josh
Rosen, Thom Neale and other Apache Spark developers improved it significantly,
most notably to add support for PyPy and Python 3.
The aim of the `cloudpickle` project is to make that work available to a wider
audience outside of the Spark ecosystem and to make it easier to improve it
further notably with the help of a dedicated non-regression test suite.