lector.types.numbers

Helpers for parsing and downcasting numeric data.

Note: Arrow uses Google’s RE2 to implement regex functionality: https://github.com/google/re2/wiki/Syntax

Classes

DecimalMode	str(object='') -> str
Downcast	Attempts truncation of floats to ints and then downcasting of ints.
Number	Attempts to parse strings into floats or ints followed by downcasting.

Functions

clean_float_pattern([thousands])	Removes characters in number strings that Arrow cannot parse.
clean_float_strings(arr, decimal)	Prepare an array of strings so that Arrow can cast the result to floats.
decimal_delimiter(s[, n_chars_max])	Infer decimal delimiter from string representation s of an input number.
infer_decimal_delimiter(arr)	Get most frequent decimal delimiter in array.
maybe_downcast_ints(arr)	Convert to smallest applicable int type.
maybe_parse_floats(arr[, threshold, decimal])	Parse valid string representations of floating point numbers.
maybe_parse_ints(arr[, threshold, allow_unsigned])	Use regex to extract castable ints.
maybe_truncate_floats(arr[, threshold])	Float to int conversion if sufficient values are kept unchanged.

Attributes

DECIMAL_CONFIDENCE_MIN
DECIMAL_SUPPORT_MIN

class lector.types.numbers.DecimalMode

Bases: str, enum.Enum

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to ‘strict’.

COMPARE = 'COMPARE'

INFER = 'INFER'

class lector.types.numbers.Downcast

Bases: lector.types.abc.Converter

Attempts truncation of floats to ints and then downcasting of ints.

convert(array)

To be implemented in subclasses.

Parameters:

array (pyarrow.Array) –

Return type:

lector.types.abc.Conversion | None

class lector.types.numbers.Number

Bases: lector.types.abc.Converter

Attempts to parse strings into floats or ints followed by downcasting.

allow_unsigned_int: bool = True

decimal: str | DecimalMode

max_int: int | None

convert(array)

To be implemented in subclasses.

Parameters:

array (pyarrow.Array) –

Return type:

lector.types.abc.Conversion | None

lector.types.numbers.clean_float_pattern(thousands=',')

Removes characters in number strings that Arrow cannot parse.

Parameters:

thousands (str) –

Return type:

str

lector.types.numbers.clean_float_strings(arr, decimal)

Prepare an array of strings so that Arrow can cast the result to floats.

Arrow allows exponential syntax and omission of 0s before and after the decimal point, i.e. the following are all valid string representations of floating point numbers: “-1e10”, “1e10”, “1e-2”, “1.2e3”, “-1.2e3”, “1.” “.12”, “-1.”, “-.1”.

Arrow doesn’t allow prefix of a positive sign indicator, nor thousands separator, i.e. the following are not(!) valid: “+1e10”, “+1.”, “+.1”, “123,456.0”

We hence remove occurrences of both the thousands character and the positive sign before extracting the floating point part of strings using regex.

Also see following for more regex parsing options: https://stackoverflow.com/questions/12643009/regular-expression-for-floating-point-numbers

Note, we don’t parse as float if there isn’t a single value with decimals. If this is the case they should be integers really, and if they haven’t been parsed as ints before, that’s because the values didn’t fit into Arrow’s largesy integer type, in which case it isn’t safe to parse as float, which Arrow would otherwise do unsafely(!) and silently.

Parameters:

• arr (pyarrow.Array) –

• decimal (str) –

Return type:

tuple[pyarrow.Array, pyarrow.Array, float]

lector.types.numbers.decimal_delimiter(s, n_chars_max=20)

Infer decimal delimiter from string representation s of an input number.

Returns None if not unambiguously inferrable.

Parameters:

• s (str) –

• n_chars_max (int) –

Return type:

str | None

lector.types.numbers.infer_decimal_delimiter(arr)

Get most frequent decimal delimiter in array.

If most frequent delimiter doesn’t occur in sufficient proportion (support), or not significantly more often than other delimiters (confidence), returns None.

Parameters:

arr (pyarrow.Array) –

Return type:

str | None

lector.types.numbers.maybe_downcast_ints(arr)

Convert to smallest applicable int type.

Parameters:

arr (pyarrow.Array) –

Return type:

pyarrow.Array | None

lector.types.numbers.maybe_parse_floats(arr, threshold=0.5, decimal=DecimalMode.INFER)

Parse valid string representations of floating point numbers.

Parameters:

• arr (pyarrow.Array) –

• threshold (float) –

• decimal (str | DecimalMode) –

Return type:

pyarrow.Array | None

lector.types.numbers.maybe_parse_ints(arr, threshold=1.0, allow_unsigned=False)

Use regex to extract castable ints.

Arrow’s internal casting from string to int doesn’t allow for an initial positive sign character, so we have to handle that separately.

Parameters:

• arr (pyarrow.Array) –

• threshold (float) –

• allow_unsigned (bool) –

Return type:

pyarrow.Array | None

lector.types.numbers.maybe_truncate_floats(arr, threshold=1.0)

Float to int conversion if sufficient values are kept unchanged.

Parameters:

• arr (pyarrow.Array) –

• threshold (float) –

Return type:

pyarrow.Array | None

lector.types.numbers.DECIMAL_CONFIDENCE_MIN = 1.5

lector.types.numbers.DECIMAL_SUPPORT_MIN = 0.2