pyarrow.OpaqueArray#

class pyarrow.OpaqueArray#

Bases: ExtensionArray

Concrete class for opaque extension arrays.

Examples

Define the extension type for an opaque array

>>> import pyarrow as pa
>>> opaque_type = pa.opaque(
...     pa.binary(),
...     type_name="geometry",
...     vendor_name="postgis",
... )

Create an extension array

>>> arr = [None, b"data"]
>>> storage = pa.array(arr, pa.binary())
>>> pa.ExtensionArray.from_storage(opaque_type, storage)
<pyarrow.lib.OpaqueArray object at ...>
[
  null,
  64617461
]

__init__(*args, **kwargs)#

Methods

`__init__`(args, *kwargs)
`buffers`(self)	Return a list of Buffer objects pointing to this array's physical storage.
`cast`(self[, target_type, safe, options, ...])	Cast array values to another data type
`copy_to`(self, destination)	Construct a copy of the array with all buffers on destination device.
`dictionary_encode`(self[, null_encoding])	Compute dictionary-encoded representation of array.
`diff`(self, Array other)	Compare contents of this array against another one.
`drop_null`(self)	Remove missing values from an array.
`equals`(self, Array other)
`fill_null`(self, fill_value)	See `pyarrow.compute.fill_null()` for usage.
`filter`(self, mask, *[, null_selection_behavior])	Select values from an array.
`from_buffers`(DataType type, length, buffers)	Construct an Array from a sequence of buffers.
`from_pandas`(obj[, mask, type])	Convert pandas.Series to an Arrow Array.
`from_storage`(BaseExtensionType typ, ...)	Construct ExtensionArray from type and storage array.
`get_total_buffer_size`(self)	The sum of bytes in each buffer referenced by the array.
`index`(self, value[, start, end, memory_pool])	Find the first index of a value.
`is_nan`(self)	Return BooleanArray indicating the NaN values.
`is_null`(self, *[, nan_is_null])	Return BooleanArray indicating the null values.
`is_valid`(self)	Return BooleanArray indicating the non-null values.
`slice`(self[, offset, length])	Compute zero-copy slice of this array.
`sort`(self[, order])	Sort the Array
`sum`(self, **kwargs)	Sum the values in a numerical array.
`take`(self, indices)	Select values from an array.
`to_numpy`(self[, zero_copy_only, writable])	Return a NumPy view or copy of this array.
`to_pandas`(self[, memory_pool, categories, ...])	Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
`to_pylist`(self, *[, maps_as_pydicts])	Convert to a list of native Python objects.
`to_string`(self, *, int indent=2, ...)	Render a "pretty-printed" string representation of the Array.
`tolist`(self)	Alias of to_pylist for compatibility with NumPy.
`unique`(self)	Compute distinct elements in array.
`validate`(self, *[, full])	Perform validation checks.
`value_counts`(self)	Compute counts of unique elements in array.
`view`(self, target_type)	Return zero-copy "view" of array as another data type.

Attributes

`device_type`	The device type where the array resides.
`is_cpu`	Whether the array is CPU-accessible.
`nbytes`	Total number of bytes consumed by the elements of the array.
`null_count`
`offset`	A relative position into another array's data.
`statistics`	Statistics of the array.
`storage`
`type`

buffers(self)#

Return a list of Buffer objects pointing to this array’s physical storage.

To correctly interpret these buffers, you need to also apply the offset multiplied with the size of the stored data type.

cast(self, target_type=None, safe=None, options=None, memory_pool=None)#

Cast array values to another data type

See pyarrow.compute.cast() for usage.

Parameters:

target_typeDataType, default None: Type to cast array to.
safebool, default True: Whether to check for conversion errors such as overflow.
optionsCastOptions, default None: Additional checks pass by CastOptions
memory_poolMemoryPool, optional: memory pool to use for allocations during function execution.

Returns:

castArray

copy_to(self, destination)#

Construct a copy of the array with all buffers on destination device.

This method recursively copies the array’s buffers and those of its children onto the destination MemoryManager device and returns the new Array.

Parameters:

destinationpyarrow.MemoryManager or pyarrow.Device: The destination device to copy the array to.

Returns:

Array

device_type#

The device type where the array resides.

Returns:

DeviceAllocationType

dictionary_encode(self, null_encoding='mask')#

Compute dictionary-encoded representation of array.

See pyarrow.compute.dictionary_encode() for full usage.

Parameters:

null_encodingstr, default “mask”: How to handle null entries.

Returns:

encodedDictionaryArray: A dictionary-encoded version of this array.

diff(self, Array other)#

Compare contents of this array against another one.

Return a string containing the result of diffing this array (on the left side) against the other array (on the right side).

Parameters:

otherArray: The other array to compare this array with.

Returns:

diffstr: A human-readable printout of the differences.

Examples

>>> import pyarrow as pa
>>> left = pa.array(["one", "two", "three"])
>>> right = pa.array(["two", None, "two-and-a-half", "three"])
>>> print(left.diff(right))

@@ -0, +0 @@ -“one” @@ -2, +1 @@ +null +”two-and-a-half”

drop_null(self)#: Remove missing values from an array.

equals(self, Array other)#

Parameters:

otherpyarrow.Array

Returns:

bool

fill_null(self, fill_value)#

See pyarrow.compute.fill_null() for usage.

Parameters:

fill_valueany: The replacement value for null entries.

Returns:

resultArray: A new array with nulls replaced by the given value.

filter(self, mask, *, null_selection_behavior='drop')#

Select values from an array.

See pyarrow.compute.filter() for full usage.

Parameters:

maskArray or array-like: The boolean mask to filter the array with.
null_selection_behaviorstr, default “drop”: How nulls in the mask should be handled.

Returns:

filteredArray: An array of the same type, with only the elements selected by the boolean mask.

static from_buffers(DataType type, length, buffers, null_count=-1, offset=0, children=None)#

Construct an Array from a sequence of buffers.

The concrete type returned depends on the datatype.

Parameters:

typeDataType: The value type of the array.
lengthint: The number of values in the array.
buffersList[Buffer | None]: The buffers backing this array.
null_countint, default -1: The number of null entries in the array. Negative value means that the null count is not known.
offsetint, default 0: The array’s logical offset (in values, not in bytes) from the start of each buffer.
childrenList[Array], default None: Nested type children with length matching type.num_fields.

Returns:

arrayArray

static from_pandas(obj, mask=None, type=None, bool safe=True, MemoryPool memory_pool=None)#

Convert pandas.Series to an Arrow Array.

This method uses Pandas semantics about what values indicate nulls. See pyarrow.array for more general conversion from arrays or sequences to Arrow arrays.

Parameters:

objndarray, pandas.Series, array-like
maskarray (bool), optional: Indicate which values are null (True) or not null (False).
typepyarrow.DataType: Explicit type to attempt to coerce to, otherwise will be inferred from the data.
safebool, default True: Check for overflows or other unsafe conversions.
memory_poolpyarrow.MemoryPool, optional: If not passed, will allocate memory from the currently-set default memory pool.

Returns:

arraypyarrow.Array or pyarrow.ChunkedArray: ChunkedArray is returned if object data overflows binary buffer.

Notes

Localized timestamps will currently be returned as UTC (pandas’s native representation). Timezone-naive data will be implicitly interpreted as UTC.

static from_storage(BaseExtensionType typ, Array storage)#

Construct ExtensionArray from type and storage array.

Parameters:

typDataType: The extension type for the result array.
storageArray: The underlying storage for the result array.

Returns:

ext_arrayExtensionArray

get_total_buffer_size(self)#

The sum of bytes in each buffer referenced by the array.

An array may only reference a portion of a buffer. This method will overestimate in this case and return the byte size of the entire buffer.

If a buffer is referenced multiple times then it will only be counted once.

index(self, value, start=None, end=None, *, memory_pool=None)#

Find the first index of a value.

See pyarrow.compute.index() for full usage.

Parameters:

valueScalar or object: The value to look for in the array.
startint, optional: The start index where to look for value.
endint, optional: The end index where to look for value.
memory_poolMemoryPool, optional: A memory pool for potential memory allocations.

Returns:

indexInt64Scalar: The index of the value in the array (-1 if not found).

is_cpu#: Whether the array is CPU-accessible.

is_nan(self)#

Return BooleanArray indicating the NaN values.

Returns:

arraybool Array

is_null(self, *, nan_is_null=False)#

Return BooleanArray indicating the null values.

Parameters:

nan_is_nullbool (optional, default False): Whether floating-point NaN values should also be considered null.

Returns:

arraybool Array

is_valid(self)#: Return BooleanArray indicating the non-null values.

nbytes#

Total number of bytes consumed by the elements of the array.

In other words, the sum of bytes from all buffer ranges referenced.

Unlike get_total_buffer_size this method will account for array offsets.

If buffers are shared between arrays then the shared portion will be counted multiple times.

The dictionary of dictionary arrays will always be counted in their entirety even if the array only references a portion of the dictionary.

null_count#

offset#

A relative position into another array’s data.

The purpose is to enable zero-copy slicing. This value defaults to zero but must be applied on all operations with the physical storage buffers.

slice(self, offset=0,