Tarantool 2.10.0

The UUID field type is now part of the SCALAR field type (gh-6042).

The UUID field type is now available in SQL. A new UUID can be generated using the new SQL built-in UUID() function (gh-5886).

[Breaking change] The timeout() method of net.box connection, marked deprecated more than four years ago (in 1.7.4), has been dropped. It negatively affected the performance of hot net.box methods, such as call() and select() if they were called without specifying a timeout (gh-6242).

Improved net.box performance by up to 70% by rewriting hot code paths in C (gh-6241).

Introduced compact tuples that allow saving 4 bytes per tuple in case of small user data (gh-5385).

Now streams and interactive transactions over streams are implemented in iproto. Every stream is associated with its ID, which is unique within one connection. All requests with the same non-zero stream ID belong to the same stream. All requests in the stream are processed synchronously. The next request will not start executing until the previous one is completed. If a request’s stream ID is 0, it does not belong to a stream and is processed in the old way.

In net.box, a stream is an object above the connection that has the same methods but allows executing requests sequentially. The ID is generated on the client-side automatically. If a user writes his own connector and wants to use streams, they must transmit the stream_id over the iproto protocol.

The primary purpose of streams is transactions via iproto. As each stream can start a transaction, several transactions can be multiplexed over one connection. There are multiple ways to begin, commit, and rollback a transaction. One can do that using the appropriate stream methods, call, eval, or execute with the SQL transaction syntax. Users can mix these methods. For example, one might start a transaction using stream:begin(), and commit it with stream:call('box.commit') or stream:execute('COMMIT').

If any request fails during the transaction, it will not affect the other requests in the transaction. If a disconnect occurs while there is an active transaction in the stream, that transaction will be rolled back if it hasn’t been committed before the connection failure (gh-5860).

Added the new memtx_allocator option to box.cfg{}. It allows selecting the appropriate allocator for memtx tuples if necessary. The possible values are system for malloc allocator and small for the default small allocator.

Implemented the system allocator based on malloc. The slab allocator, which is used for tuple allocation, has a particular disadvantage – it is prone to unresolvable fragmentation on specific workloads (size migration). In this case, the user should be able to choose another allocator. The system allocator is based on the malloc function but restricted by the same quota as the slab allocator. The system allocator does not alloc all the memory at the start. Instead, it allocates memory as needed, checking that the quota is not exceeded (gh-5419).

Added box.stat.net.thread() for reporting per thread net statistics (gh-6293).

Added the new STREAMS metric to box.stat.net. It contains statistics for iproto streams. The STREAMS contains the same counters as the CONNECTIONS metric in box.stat.net: current, RPS, and total (gh-6293).

Extended the network protocol (IPROTO) with a new request type (IPROTO_ID). It is supposed to be used for exchanging sets of supported features between the server and client (gh-6253).

Added required_protocol_version and required_protocol_features to net.box connection options. The new options allow specifying the IPROTO protocol version and features that must be supported by the server for the connection to pass (gh-6253).

[Breaking change] Added the msgpack.cfg.encode_error_as_ext configuration option to enable/disable encoding errors as MP_ERROR MsgPack extension. The option is enabled by default (gh-6433).

[Breaking change] Removed box.session.setting.error_marshaling_enabled. Error marshalling is now enabled automatically if the connector supports it (gh-6428).

Added the new REQUESTS_IN_PROGRESS and REQUESTS_IN_STREAM_QUEUE metrics to box.stat.net that contains detailed statistics for iproto requests. These metrics contain the same counters as other metrics in box.stat.net: current, RPS, and total (gh-6293).

Implemented a timeout for fiber:join in Lua (gh-6203).

Updated libev to version 4.33 (gh-4909).

Added the new box.txn_id() function. It returns the ID of the current transaction if called within a transaction, nil otherwise.

Previously, if a yield occurs for a transaction that does not support it, all its statements are rolled back but still its new statements are processed (they will roll back with each yield). Also, the transaction will be rolled back when a commit is attempted. Now we stop processing any new statements right after the first yield if a transaction does not support it.

Implemented a timeout for transactions after which they are rolled back (gh-6177).

Implemented the new C API box_txn_set_timeout function to set a timeout for transactions.

Implemented a timeout for iproto transactions after which they are rolled back (gh-6177).

Implemented the new IPROTO_TIMEOUT 0x56 key, which is used to set a timeout for transactions over iproto streams. It is stored in the body of IPROTO_BEGIN request.

Introduced box.broadcast and box.watch functions to signal/watch user-defined state changes (gh-6257).

Added watchers support to the network protocol (gh-6257).

Added watchers support to the net.box connector (gh-6257).

Now error objects with the code box.error.READONLY now have the additional fields explaining why the error happened.

Also, there is a new field box.info.ro_reason. It is nil on a writable instance, but reports a reason when box.info.ro is true (gh-5568).

Implemented the ability to open several listening sockets. In addition to the ability to pass uri as a number or string, added the ability to pass uri as a table of numbers or strings (gh-3554).

[Breaking change] net.box console support, which was marked deprecated in 1.10, has been dropped. Use require('console').connect() instead.

Added the takes_raw_args Lua function option for wrapping arguments in msgpack.object to skip decoding (gh-3349).

Implemented the graceful shutdown protocol for IPROTO connections (gh-5924).

Added fetch_schema flag to netbox.connect to control schema fetching from remote instance (gh-4789).

Added linking type (dynamic or static) to Tarantool build info.

Changed log level of some information messages from critical to info (gh-4675).

Added predefined system events: box.status, box.id, box.election, and box.schema (gh-6260).

Introduced transaction isolation levels in Lua and IPROTO (gh-6930).

Added support for backtrace feature on AARCH64 architecture (gh-6060).

Implemented collection of parent backtrace for the newly created fibers. To enable the feature, call fiber.parent_backtrace_enable. To disable it, call fiber.parent_backtrace_disable: disabled by default (gh-4302).

Introduced support for LJ_DUALNUM mode in luajit-gdb.py (gh-6224).

Introduced preliminary support of GNU/Linux ARM64 and macOS M1. In the scope of this activity, the following issues have been resolved:

• Introduced support for a full 64-bit range of lightuserdata values (gh-2712).
• Fixed memory remapping issue when the page leaves 47-bit segments.
• Fixed M1 architecture detection (gh-6065).
• Fixed variadic arguments handling in FFI on M1 (gh-6066).
• Fixed table.move misbehavior when table reallocation occurs (gh-6084).
• Fixed Lua stack inconsistency when xpcall is called with an invalid second argument on ARM64 (gh-6093).
• Fixed BC_USETS bytecode semantics for closed upvalues and gray strings.
• Fixed side exit jump target patching considering the range values of the particular instruction (gh-6098).
• Fixed current Lua coroutine restoring on an exceptional path on ARM64 (gh-6189).

Now memory profiler records allocations from traces grouping them by the trace number (gh-5814). The memory profiler parser can display the new type of allocation sources in the following format:

| TRACE [<trace-no>] <trace-addr> started at @<sym-chunk>:<sym-line>

Now the memory profiler reports allocations made by the JIT engine while compiling the trace as INTERNAL (gh-5679).

Now the memory profiler emits events of the new type when a function or a trace is created. As a result, the memory profiler parser can enrich its symbol table with the new functions and traces (gh-5815).

Furthermore, there are symbol generations introduced within the internal parser structure to handle possible collisions of function addresses and trace numbers.

Now the memory profiler dumps symbol table for C functions. As a result, memory profiler parser can enrich its symbol table with C symbols (gh-5813). Furthermore, now memory profiler dumps special events for symbol table when it encounters a new C symbol, that has not been dumped yet.

Introduced the LuaJIT platform profiler (gh-781) and the profile parser. This profiler is able to capture both host and VM stacks, so it can show the whole picture. Both C and Lua API’s are available for the profiler. Profiler comes with the default parser, which produces output in a flamegraph.pl-suitable format. The following profiling modes are available:

• Default: only virtual machine state counters.
• Leaf: shows the last frame on the stack.
• Callchain: performs a complete stack dump.

Introduced the new method table.equals. It compares two tables by value and respects the __eq metamethod.

Added support of console autocompletion for net.box objects stream and future (gh-6305).

Added the box.runtime.info().tuple metric to track the amount of memory occupied by tuples allocated on runtime arena (gh-5872).

It does not count tuples that arrive from memtx or vinyl but counts tuples created on-the-fly: say, using box.tuple.new(<...>).

[Breaking change] fiber.wakeup() in Lua and fiber_wakeup() in C became NOP on the currently running fiber.

Previously they allowed ignoring the next yield or sleep, which resulted in unexpected erroneous wake-ups. Calling these functions right before fiber.create() in Lua or fiber_start() in C could lead to a crash (in debug build) or undefined behaviour (in release build) (gh-6043).

There was a single use case for that—reschedule in the same event loop iteration which is not the same as fiber.sleep(0) in Lua and fiber_sleep(0) in C. It could be done in the following way:

in C:

fiber_wakeup(fiber_self());
fiber_yield();

in Lua:

fiber.self():wakeup()
fiber.yield()

To get the same effect in C, one can use fiber_reschedule(). In Lua, it is now impossible to reschedule the current fiber directly in the same event loop iteration. One can reschedule self through a second fiber, but it is strongly discouraged:

local self = fiber.self()
fiber.new(function() self:wakeup() end)
fiber.sleep(0)

Fixed memory leak on each box.on_commit() and box.on_rollback() (gh-6025).

Fixed the lack of testing for non-joinable fibers in fiber_join() call. This could lead to unpredictable results. Note the issue affects C level only, in Lua interface fiber:join() the protection is turned on already.

Now Tarantool yields when scanning .xlog files for the latest applied vclock and when finding the right place in .xlogs to start recovering. This means that the instance is responsive right after box.cfg call even when an empty .xlog was not created on the previous exit. Also, this prevents the relay from timing out when a freshly subscribed replica needs rows from the end of a relatively long (hundreds of MBs) .xlog (gh-5979).

The counter in x.yM rows processed log messages does not reset on each new recovered xlog anymore.

Fixed wrong type specification when printing fiber state change which led to negative fiber’s ID logging (gh-5846).

For example,

main/-244760339/cartridge.failover.task I> Instance state changed

instead of proper

main/4050206957/cartridge.failover.task I> Instance state changed

Fiber IDs were switched to monotonically increasing unsigned 8-byte integers so that there would not be IDs wrapping anymore. This allows detecting fiber’s precedence by their IDs if needed (gh-5846).

Fixed a crash in JSON update on tuple/space when it had more than one operation, they accessed fields in reversed order, and these fields did not exist. Example: box.tuple.new({1}):update({{'=', 4, 4}, {'=', 3, 3}}) (gh-6069).

Fixed invalid results produced by the json module’s encode function when it was used from Lua’s garbage collector. For instance, in functions used as ffi.gc() (gh-6050).

Added check for user input of the number of iproto threads—value must be > 0 and less than or equal to 1000 (gh-6005).

Fixed error related to the fact that if a user changed the listen address, all iproto threads closed the same socket multiple times.

Fixed error related to Tarantool not deleting the unix socket path when the work is finished.

Fixed a crash in MVCC during simultaneous update of a key in different transactions (gh-6131).

Fixed a bug when memtx MVCC crashed during reading uncommitted DDL (gh-5515).

Fixed a bug when memtx MVCC crashed if an index was created in the transaction (gh-6137).

Fixed segmentation fault with MVCC when an entire space was updated concurrently (gh-5892).

Fixed a bug with failed assertion after stress update of the same key (gh-6193).

Fixed a crash that happened when a user called box.snapshot during an incomplete transaction (gh-6229).

Fixed console client connection breakage if request times out (gh-6249).

Added missing broadcast to net.box.future:discard(). Now fibers waiting for a request result are woken up when the request is discarded (gh-6250).

box.info.uuid, box.info.cluster.uuid, and tostring(decimal) with any decimal number in Lua sometimes could return garbage if __gc handlers were used in the user’s code (gh-6259).

Fixed the error message that happened in a very specific case during MVCC operation (gh-6247).

Fixed a repeatable read violation after delete (gh-6206).

Fixed a bug when hash select{} was not tracked by MVCC engine (gh-6040).

Fixed a crash in MVCC after the drop of a space with several indexes (gh-6274).

Fixed a bug when GC at some state could leave tuples in secondary indexes (gh-6234).

Disallowed yields after DDL operations in MVCC mode. It fixes a crash which takes place in case several transactions refer to system spaces (gh-5998).

Fixed a bug in MVCC connected which happened on a rollback after DDL operation (gh-5998).

Fixed a bug when rollback resulted in unserializable behaviour (gh-6325).

At the moment, when a net.box connection is closed, all requests that have not been sent will be discarded. This patch fixes this behavior: all requests queued for sending before the connection is closed are guaranteed to be sent (gh-6338).

Fixed a crash during replace of malformed tuple into _schema system space (gh-6332).

Fixed dropping incoming messages when the connection is closed or SHUT_RDWR received and net_msg_max or readahead limit is reached (gh-6292).

Fixed memory leak in case of replace during background alter of the primary index (gh-6290).

Fixed a bug when rolled back changes appear in the built-in-background index (gh-5958).

Fixed a crash while encoding an error object in the MsgPack format (gh-6431).

Fixed a bug when an index was inconsistent after background build in case the primary index was hash (gh-5977).

Now inserting a tuple with the wrong id` field into the _priv space returns the correct error (gh-6295).

Fixed dirty read in MVCC after space alter (gh-6263, gh-6318).

Fixed a crash in case the fiber changing box.cfg.listen is woken up (gh-6480).

Fixed box.cfg.listen not reverted to the old address in case the new one is invalid (gh-6092).

Fixed a crash caused by a race between box.session.push() and closing connection (gh-6520).

Fixed a bug because of which the garbage collector could remove an xlog file that was still in use (gh-6554).

Fixed crash during granting privileges from guest (gh-5389).

Fixed an error in listening when the user passed uri in numerical form after listening unix socket (gh-6535).

Fixed a crash that could happen in case a tuple is deleted from a functional index while there is an iterator pointing to it (gh-6786).

Fixed memory leak in interactive console (gh-6817).

Fixed an assertion fail when passing a tuple without primary key fields to before_replace trigger. Now tuple format is checked before execution of before_replace triggers and after each one (gh-6780).

Banned DDL operations in space on_replace triggers, since they could lead to a crash (gh-6920).

Implemented constraints and foreign keys. Now users can create function constraints and foreign key relations (gh-6436).

Fixed a bug due to which all fibers created with fiber_attr_setstacksize() leaked until the thread exit. Their stacks also leaked except when fiber_set_joinable(..., true) was used.

Fixed a crash in MVCC related to a secondary index conflict (gh-6452).

Fixed a bug which resulted in wrong space count (gh-6421).

SELECT in RO transaction now reads confirmed data, like a standalone (autocommit) SELECT does (gh-6452).

Fixed a crash when Tarantool was launched with multiple -e or -l options without a space between the option and the value (gh-5747).

Fixed effective session and user not propagated to box.on_commit and box.on_rollback trigger callbacks (gh-7005).

Fixed usage of box.session.peer() in box.session.on_disconnect() trigger. Now it’s safe to assume that box.session.peer() returns the address of the disconnected peer, not nil, as it used to (gh-7014).

Fixed creation of a space with a foreign key pointing to the same space (gh-6961).

Fixed a bug when MVCC failed to track nothing-found range select (gh-7025).

Allowed complex foreign keys with NULL fields (gh-7046).

Added decoding of election messages: RAFT and PROMOTE to xlog Lua module (gh-6088). Otherwise tarantoolctl shows plain number in type

HEADER:
  lsn: 1
  replica_id: 4
  type: 31
  timestamp: 1621541912.4592

instead of symbolic representation

HEADER:
  lsn: 1
  replica_id: 4
  type: PROMOTE
  timestamp: 1621541912.4592

[Breaking change] Return value signedness of 64-bit time functions in clock and fiber was changed from uint64_t to int64_t both in Lua and C (gh-5989).

Fixed reversed iterators gap tracking. Instead of tracking gaps for the successors of keys, gaps for tuples shifted by one to the left of the successor were tracked (gh-7113).

Intervals received after datetime arithmetic operations may be improperly normalized if the result was negative

tarantool> date.now() - date.now()
---
- -1.000026000 seconds
...

It means that two immediately called date.now() produce very close values, which difference should be close to 0, not 1 second (gh-6882).

Fixed a bug in datetime module when date:set{tzoffset=XXX} did not produce the same result with date.new{tzoffset=XXX} for the same set of attributes passed (gh-6793).