Skip to content
Snippets Groups Projects
  1. Apr 04, 2022
    • Aleksandr Lyapunov's avatar
      txm: expose transaction isolation in iproto · a8eda574
      Aleksandr Lyapunov authored
      Introduce new option IPROTO_TXN_ISOLATION (0x59) in the body of
      IPROTO_BEGIN request, so a user can set isolation level similar
      to box.begin in lua.
      
      The value must be one of the following integers:
      enum txn_isolation_level {
      	/** Take isolation level from global default level. */
      	TXN_ISOLATION_DEFAULT,
      	/** Allow to read committed, but not confirmed changes. */
      	TXN_ISOLATION_READ_COMMITTED,
      	/** Allow to read only confirmed changes. */
      	TXN_ISOLATION_READ_CONFIRMED,
      	/** Determine isolation level automatically. */
      	TXN_ISOLATION_BEST_EFFORT,
      };
      
      Support the new option in net.box.
      
      Part of #6930
      NO_DOC=see later commits
      NO_CHANGELOG=see later commits
      a8eda574
    • Aleksandr Lyapunov's avatar
      txm: introduce transaction isolation levels · ec750af6
      Aleksandr Lyapunov authored
      Now memtx TX manager tries to determine the best isolation level
      by itself. There could be two options:
      * READ_COMMITTED, when the transaction see changes of other tx
      that are committed but not yet confirmed (written to WAL)
      * READ_CONFIRMED, when the transaction see only confirmed changes.
      
      Introduce a simple way to specify the isolation level explicitly:
      box.begin{tx_isolation = 'default'} - the same as box.begin().
      box.begin{tx_isolation = 'read-committed'} - READ_COMMITTED.
      box.begin{tx_isolation = 'read-confirmed'} - READ_CONFIRMED.
      box.begin{tx_isolation = 'best-effort'} - old automatic way.
      
      Intrduce a bit more complex but potentially faster way to set
      isolation level, like that:
      my_level = box.tnx_isolation_level.READ_COMMITTED
      ..
      box.begin{tx_isolation = my_level}
      
      For simplicity of implementation also support symmetric values as
      'READ_COMMITTED' and box.tnx_isolation_level['read-committed'].
      
      Introduce a new box.cfg option - default_tx_isolation, that is
      used as a default when a transaction is started. The option is
      dynamic and possible values are the same as in box.begin, except
      'default' which is meaningless.
      
      In addition to string value the corresponding numeric values can
      be used in both box.begin and box.cfg.
      
      Part of #6930
      NO_DOC=see later commits
      NO_CHANGELOG=see later commits
      ec750af6
    • Aleksandr Lyapunov's avatar
      txm: better detect isolation level in transaction · 4b511eeb
      Aleksandr Lyapunov authored
      When a transaction is started without specifying isolation level
      (which is impossible now) the transactional manager must choose
      the transaction level automatically, that means that is must
      detemine whether the transaction can see other prepared changes
      or not. The best effort that we can made is to check if current
      transaction is read-only or not. For read-only transactions
      there are hope and fear that it will remain read-only, and the
      best choice is not to see prepared changes. But if the transaction
      has DML statements - it must see prepared changed.
      
      Note that a read-only transaction can became read-write if it make
      a DML statement. But if a transaction ignores some other prepared
      change and then makes a DML, there are no other options except
      abort that transaction - it could not be serialized anymore.
      
      Part of #6930
      Closes #6246
      NO_DOC=see later commits
      4b511eeb
    • Aleksandr Lyapunov's avatar
      txm: remove is_prepared_ok argument from memtx_tx_tuple_clarify · 44ae54ae
      Aleksandr Lyapunov authored
      This flag actually describes isolation level:
      is_prepared_ok == true - READ_COMMITTED,
      is_prepared_ok == false - READ_CONFIRMED.
      
      Now it is always calculated as tnx != NULL.
      Let memtx_tx_tuple_clarify to calculate it by itself.
      
      No logical changes.
      
      Part of #6930
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      NO_TEST=refactoring
      44ae54ae
    • Aleksandr Lyapunov's avatar
      txm: refactor chain lookup functions · 83f8c5f5
      Aleksandr Lyapunov authored
      No logical changes.
      
      Part of #6930
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      NO_TEST=refactoring
      83f8c5f5
    • Aleksandr Lyapunov's avatar
      txm: fix a bug with wrong space:count() result · d54f4ece
      Aleksandr Lyapunov authored
      Internally space's indexes can containt dirty tuples that are
      invisible for user. That's why there's special adjustment in
      space:count() that substracts number of invisible tuple in the
      space.
      
      By a mistake that check thought that all prepared statements are
      visible, which is wrong for autocommit reads, like standalone
      space:count() without explicit transaction.
      
      Fix it by using common for all reads practice: ignore prepared
      statements if current transaction is NULL.
      
      Closes #6421
      NO_DOC=bugfix
      d54f4ece
    • Aleksandr Lyapunov's avatar
      txm: fix a crash in mvcc with secondary index conflict · c2dd8b0a
      Aleksandr Lyapunov authored
      Memtx TX manager stores a pointer to deleting statement in prepared
      story. This pointer is set in two cases:
      1. a statement deletes (or overwrites) a tuple
      2. a story becomes prepared while other inprogress TX overwrites it
      
      By design a tuple can be deleted only by primary index, the case
      when a transaction overwrites somehting in secondary index but does
      not overwrite the same tuple in primary index is prohibited. That's
      why the pointer (to deleting statement) must be set by and only by
      the next statement in the primary index chain.
      
      By mistake the pointer is set also in second index chain analysis
      after reordering which led to unexpected state of a story.
      
      The patch removes the problem.
      
      Closes #6452
      NO_DOC=bugfix
      c2dd8b0a
    • Timur Safin's avatar
      datetime: simplify boundary checks · 64faabe8
      Timur Safin authored
      We used to use very ugly and tricky approach to check that passed years,
      months and days were not exceeding supported range of values. Now we have
      introduced to `c-dt` library the new function `dt_from_ymd_checked` for
      that purpose (i.e. check that values are valid, and construct dt from
      them). So rewrite/simplify Lua code to use that entry as
      `tnt_dt_from_ymd_checked`.
      
      Part of #6731
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      64faabe8
    • Timur Safin's avatar
      datetime: huge dates support in parse functions · 5511dda7
      Timur Safin authored
      * Default parse
        - new c-dt version used which handles extended years range
          while parse relaxed iso8601 gformat strings;
        - family of functions like dt_from_ymd_checked functions
          added to the new c-dt version, now used by conversion code
          to properly handle validation of a 32-bit boundary values;
        - datetime_parse_full() modified to properly handle huge years values;
        - added tests for extended years range.
      
      * strptime-like parse
        - properly handle longer than 4 years values, negative values,
          and handle zulu suffix, which may be generated by Tarantool
          stringization routines;
      
      Part of #6731
      
      NO_DOC=internal
      NO_CHANGELOG=internal
      5511dda7
    • Timur Safin's avatar
      datetime, lua: strptime-like parse format · 02aa8f51
      Timur Safin authored
      To parse date/time strings using format string we use
      `strptime()` implementation from FreeBSD, which is
      modified to use our `struct datetime` data structure.
      
      List of supported format has been extended to include
      `%f` which is flag used whenever you need to process
      nanoseconds part of datetime value.
      
      ```
      tarantool> T = date.parse('Thu Jan  1 03:00:00 1970', {format = '%c'})
      
      tarantool> T
      - 1970-01-01T03:00:00Z
      
      tarantool> T = date.parse('12/31/2020', {format = '%m/%d/%y'})
      
      tarantool> T
      - 2020-12-31T00:00:00Z
      
      tarantool> T = date.parse('1970-01-01T03:00:00.125000000+0300',
                                {format = '%FT%T.%f%z'})
      
      tarantool> T
      - 1970-01-01T03:00:00.125+0300
      ```
      
      Part of #6731
      
      NO_DOC=internal
      NO_CHANGELOG=internal
      02aa8f51
    • Timur Safin's avatar
      datetime, lua: date parsing functions · 3c403661
      Timur Safin authored
      Datetime module provides parse function to create
      datetime object given input string.
      
      `datetime.parse` function expect 1 required argument - which is
      input string, and set of optional parameters passed as table
      in 2nd argument.
      
      Allowed attributes in this optional table are:
      * `format` - should be either 'iso8601', 'rfc3339' or `strptime`-like
        format string. [strptime format will be added as part of next
        commit];
      * `tzoffset` - to redefine offset of input string value, if there
        is no timezone provided.
      * `tz` - human-readable, Olson database, timezone identifier, e.g.
        'Europe/Moscow'. Not yet implemented in this commit.
      
      ```
      tarantool> T = date.parse('1970-01-01T00:00:00Z')
      
      tarantool> T
      - 1970-01-01T00:00:00Z
      
      tarantool> T = date.parse('1970-01-01T00:00:00',
                                {format = 'iso8601', tzoffset = 180})
      
      tarantool> T
      - 1970-01-01T00:00:00+0300
      
      tarantool> T = date.parse('2017-12-27T18:45:32.999999-05:00',
                                {format = 'rfc3339'})
      
      tarantool> T
      - 2017-12-27T18:45:32.999999-0500
      ```
      
      Implemented as per RFC https://hackmd.io/@Mons/S1Vfc_axK#%D0%AD%D1%82%D0%B0%D0%BF-3-%D0%9F%D0%B0%D1%80%D1%81%D0%B8%D0%BD%D0%B3-%D0%B4%D0%B0%D1%82-%D0%BF%D0%BE-%D1%84%D0%BE%D1%80%D0%BC%D0%B0%D1%82%D1%83
      
      Part of #6731
      
      NO_DOC=internal
      NO_CHANGELOG=internal
      3c403661
  2. Apr 01, 2022
    • Serge Petrenko's avatar
      replication: fix bootstrap failing with ER_READONLY · c1c77782
      Serge Petrenko authored
      When the master is just starting up it's possible for replica's JOIN
      request to arrive right in time to bypass ER_LOADING check (after master
      is fully recovered) but still fail due to ER_READONLY: box.cfg.read_only
      is only read and set after box_cfg() (its C part) returns.
      
      In this case the joining replica simply exits with an error and doesn't
      retry JOIN.
      
      Let's fix that. Make ER_READONLY a recoverable error and let replica
      retry joining after receiving ER_READONLY.
      
      Anonymous nodes relied on ER_READONLY to forbid replication from
      anonymous to normal replicas. That check doesn't work anymore.
      So introduce explicit checks banning replication from anonymous nodes.
      
      Note, there were some alternatives to this fix.
      
      First of all, theoretically, we could stop firing ER_LOADING later,
      after box_cfg() is complete. This solution wouldn't work because it
      would lead to deadlocks: the nodes would be stuck in replicaset_sync(),
      because each of them rejects replication with ER_LOADING.
      
      Another solution would be to read the real box.cfg.read_only value
      earlier, in order to allow replication right after the node finishes
      recovery.
      This would also be bad, because we should never let a node become
      writeable before box.cfg is finished. Even after local_recovery is
      complete, the node should stay read-only until it synchronizes with
      other replicas.
      
      That said, neither of the two alternatives fit, so the solution with
      retrying JOIN on ER_READONLY was chosen.
      
      Since the bug is fixed, re-enable the test in which it was discovered:
      replication-py/init_storage.test.py
      
      Also, remove replication/ddl.test.lua from fragile list, since this bug
      was the only reason for its fragility.
      
      Closes #5337
      Closes #6966
      
      NO_DOC=minor bugfix
      c1c77782
  3. Mar 31, 2022
  4. Mar 29, 2022
    • Vladislav Shpilevoy's avatar
      fiber: fix ignorance of flags for reused fibers · 31d27599
      Vladislav Shpilevoy authored
      fiber_new_ex() used to ignore fiber_attr flags when the fiber was
      taken from the cache, not created anew.
      
      It didn't matter much though for the public API, because the only
      public flag in fiber_attr was FIBER_CUSTOM_STACK (which can be
      set via fiber_attr_setstacksize()).
      
      Anyway that was a bug for internal API and would lead to issues in
      the future when more public flags are added. The patch fixes it.
      
      NO_DOC=Bugfix
      NO_CHANGELOG=No reproducer via public API
      31d27599
    • Vladislav Shpilevoy's avatar
      fiber: panic on cancel of a recycled fiber · dbb90274
      Vladislav Shpilevoy authored
      There was a user who complained about this code crashing:
      
          f = fiber_new_ex(...);
          fiber_start(f);
          fiber_cancel(f);
      
      The crash was at cancel. It happened because the fiber finished
      immediately. It was already recycled after fiber_start() return.
      
      Recycled fiber didn't have any flags, so fiber_cancel() didn't
      see the fiber was already dead and tried to wake it up. It crashed
      when the fiber tried to call its 'fiber->f' function which was
      NULL.
      
      In debug build the process fails earlier with an assertion on
      'fiber->fid != 0'.
      
      It can't be really fixed because the problem is the same as with
      use-after-free. The fiber could be not recycled but already freed
      completely, returned back to the mempool.
      
      This patch tries to help the users by a panic with a message
      saying that it wasn't just a crash, it is a bug in user's code.
      
      There is an alternative - make fibers never return to the mempool.
      Then fiber_cancel() could ignore recycled fibers. But it would
      lead to another problem that if the fiber is already reused, then
      fiber_cancel() would hit a totally irrelevant fiber who was
      unlucky to reuse that fiber pointer. It seems worse than panic.
      
      Same problem exists for `fiber_wakeup()`, but I couldn't figure
      out how to add a panic there and not add an `if` on the normal
      execution path (which includes 'ready' and 'running' fibers).
      
      Closes #6837
      
      NO_CHANGELOG=The same crash remains, but happens a bit earlier and
        with a message.
      
      @TarantoolBot document
      Title: `fiber_cancel()` C API clarification
      
      The documentation must warn that the fiber passed to
      `fiber_cancel()` must not be already dead unless it was set to be
      joinable. Same for `fiber_wakeup()` and all the other fiber
      functions. A dead non-joinable fiber could already be freed or
      reused.
      dbb90274
    • Vladislav Shpilevoy's avatar
      fiber: fix fibers with custom stack leak · 4ea29055
      Vladislav Shpilevoy authored
      Fibers with custom stack couldn't be reused via cord->dead list,
      but neither were ever deleted via mempool_free(). They just leaked
      until the cord was destroyed. Their custom stack also leaked.
      
      It happened for all non-joinable custom-stack fibers. That was
      because fiber_destroy() simply skipped the destruction if the
      fiber is the current one.
      
      It didn't affect joinable fibers because their fiber_destroy() is
      done in another fiber. Their stack was deleted, but the fiber
      itself still leaked.
      
      The fix makes so fiber_destroy() is never called for the current
      fiber. Instead, cord uses an approach like in pthread library -
      the fiber who wants to be deleted is saved into cord->garbage
      member. When some other fiber will want to be deleted in the
      future, it will firstly cleanup the previous one and put self into
      its place. And so on - fibers cleanup each other.
      
      The process is optimized for the case when the fiber to delete is
      not the current one - can delete it right away then.
      
      NO_DOC=Bugfix
      4ea29055
    • Igor Munkin's avatar
      lua: rewrite crc32 digest via Lua C API · 6b913198
      Igor Munkin authored
      As a result of recording <crc32:update> method or <digest.crc32>
      function wrong semantics is compiled (strictly saying, the resulting
      trace produces the different result from the one yielded by
      interpreter). The easiest solution is disabling JIT for particular
      functions, however, such approach drops the overall platform
      performance. Hence, the mentioned functions are rewritten line by line
      via Lua C API to avoid JIT misbehaviour.
      
      NO_DOC=no visible changes
      NO_CHANGELOG=no visible changes
      6b913198
    • Georgiy Lebedev's avatar
      core: fix `coro_unwcontext` invalid unwind info · c8ad49f0
      Georgiy Lebedev authored
      During the context switch required for backtracing a suspended fiber,
      unwinders go crazy, as the unwind information they had gets implicitly
      invalidated: provide an annotation for a dummy frame for
      `coro_unwcontext`, as if it were at the bottom of the call-chain — that
      way unwinders can normally proceed further.
      
      We need to know the exact location of the stack pointer: replace the
      16-byte stack alignment instruction on x86_64 macOS by adding the
      `force_align_arg_pointer` attribute to `unw_getcontext_f`.
      
      Needed for #4002
      
      NO_DOC=bug fix
      NO_CHANGELOG=bug fix
      NO_TEST=unwind information annotation in inline assembly
      c8ad49f0
    • cha-cha369's avatar
      docs: fix typos in module.h · 1afde72c
      cha-cha369 authored
      NO_DOC=no behavior changes
      NO_TEST=no behavior changes
      NO_CHANGELOG=no behavior changes
      1afde72c
    • Yan Shtunder's avatar
      net.box: add predefined system events for pub/sub · e1d2f7f0
      Yan Shtunder authored
      Added predefined system events: box.status, box.id, box.election and
      box.schema.
      
      Closes #6260
      
      @TarantoolBot document
      Title: Built-in events for pub/sub
      
      Built-in events are needed, first of all, in order to learn who is the
      master, unless it is defined in an application specific way. Knowing who
      is the master is necessary to send changes to a correct instance, and
      probably make reads of the most actual data if it is important. Also
      defined more built-in events for other mutable properties like leader
      state change, his election role and election term, schema version change
      and instance state.
      
      Built-in events have a special naming schema - their name always starts
      with box.. The prefix is reserved for built-in events. Creating new events
      with this prefix is banned. Below is a list of all the events + their names
      and values:
      
      1. box.id
      Description - identification of the instance. Changes are extra rare. Some
      values never change or change only once. For example, instance UUID never
      changes after the first box.cfg. But is not known before box.cfg is called.
      Replicaset UUID is unknown until the instance joins to a replicaset or
      bootsa new one, but the events are supposed to start working before that -
      right at listen launch. Instance numeric ID is known only after
      registration. On anonymous replicas is 0 until they are registered
      officially.
      Value - {
          MP_STR “id”: MP_UINT; box.info.id,
          MP_STR “instance_uuid”: MP_UUID; box.info.uuid,
          MP_STR “replicaset_uuid”: MP_UUID box.info.cluster.uuid,
      }
      
      2. box.status
      Description - generic blob about instance status. Its most commonly used
      and not frequently changed config options and box.info fields.
      Value - {
          MP_STR “is_ro”: MP_BOOL box.info.ro,
          MP_STR “is_ro_cfg”: MP_BOOL box.cfg.read_only,
          MP_STR “status”: MP_STR box.info.status,
      }
      
      3. box.election
      Description - all the needed parts of box.info.election needed to find who
      is the most recent writable leader.
      Value - {
          MP_STR “term”: MP_UINT box.info.election.term,
          MP_STR “role”: MP_STR box.info.election.state,
          MP_STR “is_ro”: MP_BOOL box.info.ro,
          MP_STR “leader”: MP_UINT box.info.election.leader,
      }
      
      4. box.schema
      Description - schema-related data. Currently it is only version.
      Value - {
          MP_STR “version”: MP_UINT schema_version,
      }
      
      Built-in events can't be override. Meaning, users can't be able to call
      box.broadcast(‘box.id’, any_data) etc.
      
      The events are available from the very beginning as not MP_NIL. It's
      necessary for supported local subscriptions. Otherwise no way to detect
      whether an event is even supported at all by this Tarantool version. If
      events are broadcast before box.cfg{}, then the following values will
      available:
          box.id = {}
          box.schema = {}
          box.status = {}
          box.election = {}
      
      This way the users will be able to distinguish an event being not supported
      at all from box.cfg{} being not called yet. Otherwise they would need to
      parse _TARANTOOL version string locally and peer_version in netbox.
      
      Example usage:
      
       * Client:
         ```lua
         conn = net.box.connect(URI)
         -- Subscribe to updates of key 'box.id'
         w = conn:watch('box.id', function(key, value)
             assert(key == 'box.id')
             -- do something with value
         end)
         -- or to updates of key 'box.status'
         w = conn:watch('box.status', function(key, value)
             assert(key == 'box.status')
             -- do something with value
         end)
         -- or to updates of key 'box.election'
         w = conn:watch('box.election', function(key, value)
             assert(key == 'box.election')
             -- do something with value
         end)
         -- or to updates of key 'box.schema'
         w = conn:watch('box.schema', function(key, value)
             assert(key == 'box.schema')
             -- do something with value
         end)
         -- Unregister the watcher when it's no longer needed.
         w:unregister()
         ```
      e1d2f7f0
  5. Mar 28, 2022
    • Vladimir Davydov's avatar
      memtx: drop UNCHANGED (get = get_raw) index vtab optimization · 5e340b6e
      Vladimir Davydov authored
      We use a special, less efficient index vtab if a space can store
      compressed tuples. The problem is it isn't enough to look at a space
      definition to figure out if there are compressed tuples in the space:
      there may be compressed tuples left from before the alter operation that
      disabled compression, since we don't rebuild tuples on alter. To update
      an index vtab dynamically, we implement some complicated logic, but
      it's buggy (results in a test failure in EE). Fixing it requires some
      non-trivial effort, because a vtab may be changed after index creation
      (when a space format is updated).
      
      Let's drop this optimization altogether for now and use the same vtab
      for both compressed and uncompressed indexes. We might return to this
      issue in future, but first we need to run some benchmarks to check if
      this optimization is worth the complexity. Possible ways how we could
      resurrect this optimization:
       - Call get_raw from get directly (without function pointer), inline
         memtx_prepare_result_tuple, and move is_compressed flag to struct
         tuple for better cache locality.
       - Rebuild all tuples on space alter and use a different vtab for
         compressed indexes.
      
      NO_DOC=bug fix
      NO_TEST=enterprise
      NO_CHANGELOG=unrelased
      5e340b6e
    • Vladimir Davydov's avatar
      memtx: fix assertion in memtx_tx_history_rollback_stmt · c03c34e9
      Vladimir Davydov authored
      If tuple compression is enabled, txn_stmt::new_tuple points to
      a temporary tuple created by uncompressing a compressed tuple stored
      in an index. We must use txn_stmt::rollback_info::new_tuple instead.
      
      NO_DOC=bug fix
      NO_TEST=enterprise
      NO_CHANGELOG=unreleased
      c03c34e9
  6. Mar 24, 2022
    • Vladimir Davydov's avatar
      say: move log_vsay to header · 33e04a9e
      Vladimir Davydov authored
      We need it for audit log.
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      33e04a9e
    • Aleksandr Lyapunov's avatar
      box: implement complex foreign keys · 1150adf2
      Aleksandr Lyapunov authored
      Implement complext foreign keys addition to field foreign keys.
      They are quite similar to field foreign keys, the difference is:
      * The are set in space options instead of format field definition.
      * Several fields may be specified in relation.
      * By design field foreign keys are more optimal.
      
      One can set up foreign keys in space options:
      box.schema.space.create(.. {.., foreign_key=<foreign_key>})
      where foreign_key can be of one of the following forms:
       foreign_key={space=..,field=..}
       foreign_key={<name1>={space=..,field=..}, ..}
      where field must be a table with local -> foreing fields mapping:
       field={local_field1=foreign_field1, ..}
      
      NO_DOC=see later commits
      NO_CHANGELOG=see later commits
      1150adf2
    • Aleksandr Lyapunov's avatar
      box: implement field foreign keys · d950fdde
      Aleksandr Lyapunov authored
      Foreign key is a special type of constraint that makes a relation
      between spaces. When declared in some space, each tuple in that
      space refers to some tuple in another, foreign space. Reference is
      defined in foreign key definition as a correspondence of field of
      that spaces, local and remote.
      
      Foreign key preserves that reference between tuples and consists
      of two checks:
      1. When a tuple is added to space with foreign space constraint,
      it must be checked that there is a corresponding tuple in foreign
      space, with the same values in fields according to foreign key
      definitiion.
      2. When a tuple is deleted from space that is a foreign space for
      some other space, it must be checked that no tuple references the
      deleted one.
      
      This commit introduces field foreign keys that link spaces by
      just one field. They are declared in tuple format in one of the
      following forms:
       space:format{..,{name=.., foreign_key=<fkey>},..}
       space:format{..,{name=.., foreign_key={<name>=<fkey>}},..}
      Where fkey has a form of a table:
       {space=<foreign space id/name>, field=<foreign field id/name>}
      
      NO_DOC=see later commits
      NO_CHANGELOG=see later commits
      d950fdde
    • Aleksandr Lyapunov's avatar
      box: add pin/unping infrastructure for spaces · b00a4579
      Aleksandr Lyapunov authored
      There are cases when we need to be sure that a space by given
      id or name is not deleted; and if it is replaced (in space cache),
      there's need to track pointer to new space. Like ib constraints:
      they must hold a pointer to struct space while it's very hard to
      determine whether there'a constraint that points to given space.
      
      Implement space pin/unpin for this purpose. You can pin a space to
      declare that the space is require to exist. To have to unpin it
      when the space is not needed anymore.
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      NO_TEST=refactoring
      b00a4579
    • Aleksandr Lyapunov's avatar
      box: move space_cache to a separate file · 9b1c1e8d
      Aleksandr Lyapunov authored
      I'm going to extend space cache API so it should be separated.
      One function went to space.h/c.
      No logical changes.
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      NO_TEST=refactoring
      9b1c1e8d
    • Aleksandr Lyapunov's avatar
      box: use field names in tuple constraint function · 1d35d866
      Aleksandr Lyapunov authored
      The previous commit adds tuple constraint lua functions that check
      format of entire tuple. The problem was that in those functions
      tuple could be accessed only by field indexes.
      
      Add an ability to use field names too.
      
      NO_DOC=see later commits
      NO_CHANGELOG=see later commits
      1d35d866
    • Aleksandr Lyapunov's avatar
      box: implement tuple constraints · 53f5d4e7
      Aleksandr Lyapunov authored
      Implement whole tuple constraints in addition to field constraints.
      They are quite similar to field constraints, the difference is:
       * The are set in space options instead of format field definition.
       * Entire tuple is passed to check function.
       * By design field constraints are a bit more optimal.
      
      One can set up constraint in space options, with one or several
      functions that must be present in _func space:
      box.schema.space.create(.. {.. constraint='func1'})
      box.schema.space.create(.. {.. constraint={name1='func1'})
      box.schema.space.create(.. {.. constraint={name1='f1', name2='f2'})
      
      NO_DOC=see later commits
      NO_CHANGELOG=see later commits
      53f5d4e7
    • Aleksandr Lyapunov's avatar
      box: introduce a pair of tuple_format_new helpers · 4b8dc6b7
      Aleksandr Lyapunov authored
      tuple_format_new has lots of arguments, all of them necessary
      indeed. But a small analysss showed that almost always there are
      only two kinds of usage of that function: with lots of zeros as
      arguments and lots of values taken from space_def.
      
      Make two versions of tuple_format_new:
      simple_tuple_format_new, with all those zeros omitted, and
      space_tuple_format_new, that takes space_def as an argument.
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      4b8dc6b7
    • Aleksandr Lyapunov's avatar
      box: implement field constraints · ed9b982d
      Aleksandr Lyapunov authored
      Introduce field constraints - limitaions for particular fields.
      Each constraint must refer to a function in _func space. For the
      first step we expect lua functions with body there.
      
      Field constraint checks are very close to field type checks, so
      it's natural to implement them in tuple formats. On the other hand
      tuple formats belong to tuple library, that does not include
      functions (func.h/c etc), so constraints are split into two parts:
      - a part in tuple library that implements arbitrary constraints
       with pointers to functions that actually check constraints.
      - a part in box library which uses the part above, sets particular
       check functions and handles alter etc.
      
      There are two not-so-obvious problems that are solved here:
       - Functions in _func space must be preserved while used by such
       constraints. Func pinning is used for this purpose.
       - During initial recovery constraits are created before _func
       space recovery, so we have to pospone constraint initialization.
      
      One can set up constraint for any field in tuple format with one
      or several functions that must be present in _func space:
      space:format{name='a', constraint='func1'}
      space:format{name='a', constraint={name1='func1'}}
      space:format{name='a', constraint={name1='func1', name2='func2'}}
      
      So constraint(s) can be set by one function name or by lua table
      with function name values. Each consraints has a name that can be
      specified directly (with string key in table) or imlicitly set to
      the name of function.
      
      The check function receives two arguments: the checking value and
      the name of the constraint. Also the name of the failed constraint
      is present in raised exception.
      
      The only way to pass the constraint is to return true from its
      function. All other values and exception are treated as failure
      (exeptions are also logged).
      
      NO_DOC=see later commits
      NO_CHANGELOG=see later commits
      ed9b982d
    • Aleksandr Lyapunov's avatar
      box: implement ability to add a string to C port. · 44a49408
      Aleksandr Lyapunov authored
      Now C port allows to add a tuple or raw msgpack to it.
      
      Add another function that encodes and appends given string.
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      NO_TEST=refactoring
      44a49408
    • Aleksandr Lyapunov's avatar
      salad: introduce group alloc · af7667d1
      Aleksandr Lyapunov authored
      gpr_alloc is a small library that is designed for simplification
      of allocation of several objects in one memory block. It could be
      anything, but special attention is given to string objects, that
      are arrays of chars.
      
      Typical usage consist of two phases: gathering total needed size
      of memory block and creation of objects in given block.
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      af7667d1
    • Aleksandr Lyapunov's avatar
      box: intoduce engine recovery state · f7435757
      Aleksandr Lyapunov authored
      There's several important stages of recovery of database: loading
      from snapshot, then loading from WAL(s) and then normal operation.
      
      Introduce a global recovery state that shows this stage.
      
      Note that there's already a recovery state in memtx engine which is
      very close but still different to the new introduced state. That
      state in memtx is a private property of memtx that internally shows
      initialization status of memtx spaces and indexes. Memtx can set the
      value for convenience' sake, for example it can jump directly to
      MEMTX_OK before snapshot loading in case of force recovert.
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      NO_TEST=refactoring
      f7435757
    • Aleksandr Lyapunov's avatar
      box: add OPT_CUSTOM to opt_def · 2a88b535
      Aleksandr Lyapunov authored
      opt_def is an option parser from msgpack by given scheme.
      The scheme consists of set of predefined types that the parser
      can handle (something like int, enum, str). The problen is that
      those predefined types must be generic, but there are cases when
      a specific unusual must be parsed.
      
      This patch introduces OPT_CUSTOM type that uses arbitrary function
      callback and can be used for any non-generic option value.
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      NO_TEST=refactoring
      2a88b535
    • Aleksandr Lyapunov's avatar
      box: add pin/unping infrastructure for func cache · ffc9cee4
      Aleksandr Lyapunov authored
      There are cases when we need to be sure that a function is not
      deleted and/or removed from func cache. For example constraints:
      they must hold a pointer to struct func while it's very hard to
      determine whether there'a constraint that points to given func.
      
      Implement func pin/unpin for this purpose. You can pin a func to
      declare that the func must not be deleted. To have to unpin it
      when the func is not needed anymore.
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      ffc9cee4
    • Aleksandr Lyapunov's avatar
      lib: refactor assoc library a bit · 259a7584
      Aleksandr Lyapunov authored
      - Use uint32_t for string length. Actually internally it cannot
      take more that INT_MAX length, so uin32_t is enough. This change
      makes the hash table a bit more compact.
      - Rename mh_strnptr_find_inp -> mh_strnptr_find_str. I beleive it
      makes it more understandable.
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      NO_TEST=refactoring
      259a7584
    • Aleksandr Lyapunov's avatar
      box: move func_cache to a separate file · 3cad9398
      Aleksandr Lyapunov authored
      I'm going to extend func cache API so it should be separated.
      A couple of comments added.
      No logical changes.
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      NO_TEST=refactoring
      3cad9398
    • Aleksandr Lyapunov's avatar
      box: trashify port data after destruction · bdd821bf
      Aleksandr Lyapunov authored
      Fill port with a corrupted data in debug (TRASH it) in order to
      detect double port destruction early.
      
      Add a comment for func_call function that describes what states
      of ports are expected before and after func_call.
      
      Fix port usage in lua using ffi - allocate a full port structure
      instead of a (bit smaller) structure port_c. That makes any port
      structure to have fixed determined size which in turn makes safe
      to cast and use different port types.
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      NO_TEST=refactoring
      bdd821bf
    • Aleksandr Lyapunov's avatar
      sql: initialize field def properly · eb09d3c7
      Aleksandr Lyapunov authored
      There must be a normal code flow: if you want to initialize some
      structure you should set members you want to and set with defaults
      the rest members. Such a code flow would be durable when some new
      members are added to the struct.
      
      Make field_def structure to comply those rules.
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      NO_TEST=refactoring
      eb09d3c7
Loading