TLDR; Creating a relationship

Owned by Matthew Iverson, created
Oct 21, 2020
4 min read

Relationships are a fundamental part of Legion. It connect to completely independent functions/services that otherwise don’t integrate

 

Requirements

  1. LEXs

    1. lex-conditioner

    2. lex-tasker

    3. lex-transformer

    4. lex-lex

  2. Legion::Data to be connected and have access to a database. Relationships are stored in a table so without the table, you can’t have a relationship entry

Instructions, Direct Database

  1. Ensure that the LEXs you want to use have been installed and are registered inside the extensions table(this requires you to run the lex-lex extension

  2. Add a new row inside the relationships data

    1. Required Columns

      1. active(Boolean, default:true): Is this relationship active? If 0, then it won’t create a task for the trigger

      2. trigger_id(Integer): A FK to the functions table to tell you what function triggers this relationship to run

      3. action_id(Integer): A FK to the functions table to tell you what function runs after this relationship is triggered

      4. delay(Integer, default:0): Should we delay execution of this task? A positive number is number of seconds to wait to run. lex-tasker must be running otherwise delayed tasks will not be grabbed

      5. allow_new_chains(Boolean, default: true): Can this trigger a new chain(injection) or must it be apart of the chain_id defined above

    2. Optional Columns

      1. name(String): the name/description of the relationship. Only used to inform users

      2. chain_id(Integer): A FK to the chains table allowed you to create relationships that are connected

      3. conditions(Text): Conditions which must pass validation in order to send this task to the action

        1. If this is null, empty or < 4 characters long, it will skip the conditioner step automatically and go straight to transformer

        2. See -___- for conditioner examples

      4. transformation(Text): A definition on how to manipulate the data into a format the action function can use

        1. If this is null, empty or < 4 characters long, it will skip the transformer step and automatically go to the output task queue for the action_function

        2. See -___- for transformer examples

      5. debug(Boolean, default:false): Should we add extra task logs for this event. Warning this will add significantly more entries into the last_logs table

  3. lex-tasker is the extensions that checks for a subtask. Without it, the check subtask queue will begin to fill and relationships won’t be checked

  4. tasker automatically checks for relationships based on the dataset. It is however, cached for either 5 seconds or whatever the cache TTL setting is set to. The reason for this is that if you have hundreds of tasks running, it can get extremely expensive to find subtasks for every task that completes. Allowing this to be cached dramatically increases performance

  5. find_subtask settings

    1. extensions.tasker.find_subtasks_cache_ttl default = 5

    2. extensions.tasker.find_subtasks_cache default = true

    3. extensions.tasker.find_trigger_cache_ttl default = 60

    4. extensions.tasker.find_trigger_cache default = true

  6. fetch_delayed settings

    1. extensions.tasker.fetch_delayed.interval default = 1, it’s worth noting that each worker of tasker will push a refresh message at this interval

      1. 1 instance, 1 tasker worker, interval = 1, you should see 1 message per second in the task.fetch_delayed queue

      2. 2 instances, 4 tasker workers, interval = 2, you should see ~4 messages per second in the task.fetch_delayed queue

      3. It’s worth noting that the fetch_delayed queue uses the x-single-active-consumer flag so you don’t have to worry about race conditions and double task queueing

 

Task Path

  1. task completes sending a message to the task.subtask.check queue

  2. tasker grabs all rows out of the relationships table where this function is the trigger

  3. it skips all tasks that are not active or don’t meet the chain requirements

  4. if delay is set to 0, skip to step 6

  5. task status is updated to task.delayed to be picked up by fetch_delayed worker once enough time has passed

  6. tasker parses all the things for this relationship row including where is this thing going(the action function)

  7. If there are no conditions and

    1. there is a transformation defined, skip to step 9

    2. there is no transformation defined, skip to step 10

  8. Conditioner checks the conditions

    1. if it fails checks, then update status to conditioner.failed and process ends

    2. if it succeeds and there is not a transformation, skip to step

    3. if it succeeds, continue

  9. transform picks up the task, and runs the transformer engine to insert variables, modify the payload, etc with whatever you defined in the transformation column

  10. task is sent to the queue that the worker users. Each function is associated with a runner. Each runner has a queue name. The task will be sent to the queue with the queue name and routing key will = extension.runner.function

  11. worker picks up work, completes it and sends a message to task.subtask.check to see if there are any relationships for tasker to create work for(repeat the steps)

 

graph LR tasker_check_subtask[Tasker Check Subtask] tasker_fetch_delayed[Tasker Fetch Delayed] conditioner[Lex-Conditioner] transformer[Lex-transformer] worker[LEX Worker for action function] dec_1{Are there conditions defined?} dec_2{Transformation defined?} tasker_check_subtask --No Delay--> dec_1 tasker_check_subtask --Delayed--> tasker_fetch_delayed tasker_fetch_delayed -.-> dec_1 dec_1 --Yes--> conditioner dec_1 --No--> dec_2 conditioner -.Conditions Passed.-> dec_2 conditioner -.Conditions Failed.-> end_path dec_2 --Yes--> transformer dec_2 --No--> worker transformer -.-> worker

Related articles