Fix decimal values in vinyl spaces when upgrading to 2.10.1

This is an upgrade guide for fixing one specific problem which could happen with decimal values in vinyl spaces. It’s only relevant when you’re upgrading from Tarantool version <= 2.10.0 to anything >= 2.10.1.

Before gh-6377 was fixed, decimal and double values in a scalar or number index could end up in the wrong order after the update. If such an index has been built for a space that uses the vinyl storage engine, the index is persisted and is not rebuilt even after the upgrade. If this is the case, the user has to rebuild the affected indexes manually.

Here are the rules to determine whether your installation was affected. If all of the statements listed below are true, you have to rebuild indexes for the affected vinyl spaces manually.

You were running Tarantool version 2.10.0 and below.
You have spaces with the vinyl storage engine.
The vinyl spaces have number or scalar indexes.
The tuples in these spaces may contain both decimal and double Inf or NaN values.

If this is the case for you, you can run the following script, which will find all the affected indices:

local fiber = require('fiber')
local decimal = require('decimal')

local function isnan(val)
    return type(val) == 'number' and val ~= val
end

local function isinf(val)
    return val == math.huge or val == -math.huge
end

local function vinyl(id)
    return box.space[id].engine == 'vinyl'
end

require_rebuild = {}
local iters = 0
for _, v in box.space._index:pairs({512, 0}, {iterator='GE'}) do
    local id = v[1]
    iters = iters + 1
    if iters % 1000 == 0 then
        fiber.yield()
    end
    if vinyl(id) then
        local format = v[6]
        local check_fields = {}
        for _, fmt in pairs(v[6]) do
            if fmt[2] == 'number' or fmt[2] == 'scalar' then
                table.insert(check_fields,  fmt[1] + 1)
            end
        end
        local have_decimal = {}
        local have_nan = {}
        if #check_fields > 0 then
            for k, tuple in box.space[id]:pairs() do
                for _, i in pairs(check_fields) do
                    iters = iters + 1
                    if iters % 1000 == 0 then
                        fiber.yield()
                    end
                    have_decimal[i] = have_decimal[i] or
                                    decimal.is_decimal(tuple[i])
                    have_nan[i] = have_nan[i] or isnan(tuple[i]) or
                                isinf(tuple[i])
                    if have_decimal[i] and have_nan[i] then
                        table.insert(require_rebuild, v)
                        goto out
                    end
                end
            end
        end
    end
    ::out::
end

The indices requiring a rebuild will be stored in the require_rebuild table. If the table is empty, you’re safe and can continue using Tarantool as before.

If the require_rebuild table contains some entries, you can rebuild the affected indices with the following script.

Note

Please run the script below only on the master node and only after all the nodes are upgraded to the new Tarantool version.

local log = require('log')

local function rebuild_index(idx)
    local index_name = idx[3]
    local space_name = box.space[idx[1]].name
    log.info("Rebuilding index %s on space %s", index_name, space_name)
    if (idx[2] == 0) then
        log.error("Cannot rebuild primary index %s on space %s. Please, "..
                "recreate the space manually", index_name, space_name)
        return
    end
    log.info("Deleting index %s on space %s", index_name, space_name)
    local v = box.space._index:delete{idx[1], idx[2]}
    if v == nil then
        log.error("Couldn't find index %s on space %s", index_name, space_name)
        return
    end
    log.info("Done")
    log.info("Creating index %s on space %s", index_name, space_name)
    box.space._index:insert(v)
end

for _, idx in pairs(require_rebuild) do
    rebuild_index(idx)
end

The script might fail on some of the indices with the following error: “Cannot rebuild primary index index_name on space space_name. Please, recreate the space manually”. If this happens, automatic index rebuild is impossible, and you have to manually re-create the space to ensure data integrity:

Create a new space with the same format as the existing one.
Define the same indices on the freshly created space.
Iterate over the old space’s primary key and insert all the data into the new space.
Drop the old space.

Version:

Fix decimal values in vinyl spaces when upgrading to 2.10.1