Evolution of Netflix Conductor
                     Evolution of Netflix 
Conductor
Conductor is a workflow orchestration engine developed and open-sourced by 
Netflix. If you’re new to Conductor, this earlier blog post and the 
documentation should help you get started and acclimatized to Conductor.
Over the most recent a long time since commencement, Conductor has seen 
wide appropriation and is instrumental in running various center work 
processes at Netflix. A considerable lot of the Netflix Content and Studio 
Engineering administrations depend on Conductor for proficient handling of 
their business streams. The Netflix Media Database (NMDB) is one such 
model. 
In this blog, we might want to show the most recent updates to Conductor, 
address a portion of the much of the time posed inquiries and thank the 
network for their commitments.
Visit here for more details office setup 
today
How we’re using Conductor at Netflix
1.Deployment
Conductor is one of the most heavily used services within Content Engineering at Netflix. 
Of the multitude of modules that can be plugged into Conductor as shown in the image 
below, we use the Jersey server module, Cassandra for persisting execution data,Dynomite 
for persisting metadata, Dyno Queues as the queuing recipe built on top of Dynomite, 
Elastic search as the secondary data store and indexer, and Netflix Spectator + Atlas for 
Metrics. Our cluster size ranges from 12–18 instances of AWS EC2 m4.4xlarge instances, 
typically running at ~30% capacity.
We do not maintain an internal fork of Conductor within Netflix. Instead, we use a 
wrapper that pulls in the latest version of Conductor and adds Netflix infrastructure 
components and libraries before deployment. This allows us to proactively push changes to 
the open source version while ensuring that the changes are fully functional and well-
tested.
2. Adoption
As of writing this blog, Conductor orchestrates 600+ workflow definitions owned by 50+ 
teams across Netflix. While we’re not (yet) actively measuring the nth percentiles, our 
production workloads speak for Conductor’s performance. Below is a snapshot of our 
Kibana dashboard which shows the workflow execution metrics over a typical 7-day period.
Use Cases
Some of the use cases served by Conductor at Netflix can be categorized under:
• Content Ingest and Delivery
• Content Quality Control
• Content Localization
• Encodes and Deployments
• IMF Deliveries
• Marketing Tech
• Studio Engineering
What’s New
gRPC Framework
One of the key features in v2.0 was the introduction of the gRPC 
framework as an alternative/auxiliary to REST. This was contributed by 
our counterparts at GitHub, thereby strengthening the value of 
community contributions to Conductor.
Cassandra Persistence Layer
To enable horizontal scaling of the data store for large volume of concurrent 
workflow executions (millions of workflows/day), Cassandra was chosen to provide 
elastic scaling and meet throughput demands.
External Payload Storage
External payload storage was implemented to prevent the usage of Conductor as a 
data persistence system and to reduce the pressure on its backend data store.
Dynamic Workflow Executions
For use cases where the need arises to execute a large/arbitrary number of varying 
workflow definitions or to run a one-time ad hoc workflow for testing or analytical 
purposes, registering definitions first with the metadata store in order to then 
execute them only once, adds a lot of additional overhead. The ability to 
dynamically create and execute workflows removes this friction. This was another 
great addition that stemmed from our collaboration with GitHub.
Workflow Status Listener
Conductor can be configured to publish notifications to external systems or queues 
upon completion/termination of workflows. The workflow status listener provides 
hooks to connect to any notification system of your choice. The community has 
contributed an implementation that publishes a message on a dyno queue based on the 
status of the workflow. An event handler can be configured on these queues to trigger 
workflows or tasks to perform specific actions upon the terminal state of the workflow.
Bulk Workflow Management
There has always been a need for bulk operations at the workflow level from an 
operability standpoint. When running at scale, it becomes essential to perform 
workflow level operations in bulk due to bad downstream dependencies in the worker 
processes causing task failures or bad task executions. Bulk APIs enable the operators 
to have macro-level control on the workflows executing within the system.
Decoupling Elastic search from Persistence
This inter-dependency was removed by moving the indexing layer into separate 
persistence modules, exposing a property (workflow.elasticsearch.instanceType) to 
choose the type of indexing engine. Further, the indexer and persistence layer have 
been decoupled by moving this orchestration from within the primary persistence layer 
to a service layer through the Execution DAOFacade.
ES5/6 Support
Support for Elastic search versions 5 and 6 have been added as part of the major 
version upgrade to v2.x. This addition also provides the option to use the Elastic 
search Rest Client instead of the Transport Client which was enforced in the 
previous version. This opens the route to using a managed Elastic search cluster (a 
la AWS) as part of the Conductor deployment.
Task Rate Limiting & Concurrent Execution Limits
Task rate limiting helps achieve bounded scheduling of tasks. The task definition 
parameter rateLimitFrequencyInSeconds sets the duration window, 
while rateLimitPerFrequency defines the number of tasks that can be scheduled 
in a duration window. On the other hand, concurrentExecLimitprovides 
unbounded scheduling limits of tasks. I.e the total of current scheduled tasks at 
any given time will be under concurrentExecLimit. The above parameters can be 
used in tandem to achieve desired throttling and rate limiting.
API Validations
Validation was one of the core features missing in Conductor 1.x. To improve 
usability and operability, we added validations, which in practice has greatly helped 
find bugs during creation of workflow and task definitions. Validations enforce the 
user to create and register their task definitions before registering the workflow 
definitions using these tasks. It also ensures that the workflow definition is well-
formed with correct wiring of inputs and outputs in the various tasks within the 
workflow. Any anomalies found are reported to the user with a detailed error 
message describing the reason for failure.
Developer Labs, Logging and Metrics
We have been continually improving logging and metrics, and revamped the 
documentation to reflect the latest state of Conductor. To provide a smooth on 
boarding experience, we have created developer labs, which guides the user 
through creating task and workflow definitions, managing a workflow lifecycle, 
configuring advanced workflows with eventing etc., and a brief introduction to 
Conductor API, UI and other modules.

New Task Types
System tasks have proven to be very valuable in defining the Workflow structure and 
control flow. As such, Conductor 2.x has seen several new additions to System tasks, 
mostly contributed by the community:
Lambda
Lambda Task executes ad-hoc logic at Workflow run-time, using the Nashorn Javascript 
evaluator engine. Instead of creating workers for simple evaluations, Lambda task enables 
the user to do this inline using simple Javascript expressions.
Terminate
Terminate task is useful when workflow logic should terminate with a given output. For 
example, if a decision task evaluates to false, and we do not want to execute remaining 
tasks in the workflow, instead of having a DECISION task with a list of tasks in one case 
and an empty list in the other, this can scope the decide and terminate workflow 
execution.
ExclusiveJoin
Exclusive Join task helps capture task output from a DECISION task’s flow. This is useful 
to wire task inputs from the outputs of one of the cases within a decision flow. This 
data will only be available during workflow execution time and the ExclusiveJoin task 
can be used to collect the output from one of the tasks in any of decision branches.
For in-depth implementation details of the new additions, please refer 
the documentation.
What’s next
There are a lot of features and enhancements we would like to add to Conductor. The 
below wish list could be considered as a long-term road map. It is by no means exhaustive, 
and we are very much welcome to ideas and contributions from the community. Some of 
these listed in no particular order are:
Advanced Eventing with Event Aggregation and Distribution
At the moment, event generation and processing is a very simple implementation. An event 
task can create only one message, and a task can wait for only one event.
We envision an Event Aggregation and Distribution mechanism that would open up 
Conductor to a multitude of use-cases. A coarse idea is to allow a task to wait for 
multiple events, and to progress several tasks based on one event.
UI Improvements
While the current UI provides a neat way to visualize and track workflow executions, 
we would like to enhance this with features like:
• Creating metadata objects from UI
• Support for starting workflows
• Visualize execution metrics
• Admin dashboard to show outliers
New Task types like Goto, Loop etc.
Conductor has been using a Directed Acyclic Graph (DAG) structure to define a 
workflow. The Goto and Loop on tasks are valid use cases, which would deviate from 
the DAG structure. We would like to add support for these tasks without violating 
the existing workflow execution rules. This would help unlock several other use 
cases like streaming flow of data to tasks and others that require repeated execution 
of a set of tasks within a workflow.
Support for reusable commonly used tasks like Email, DatabaseQuery etc.
Similarly, we’ve seen the value of shared reusable tasks that does a specific thing. 
At Netflix internal deployment of Conductor, we’ve added tasks specific to 
services that users can leverage over recreating the tasks from scratch. For 
example, we provide a TitusTask which enables our users to launch a new Titus 
container as part of their workflow execution.
We would like to extend this idea such that Conductor can offer a repository of 
commonly used tasks.
Push based task scheduling interface
Current Conductor architecture is based on polling from a worker to get tasks that 
it will execute. We need to enhance the grpc modules to leverage the 
bidirectional channel to push tasks to workers as and when they are scheduled, 
thus reducing network traffic, load on the server and redundant client calls.
Validating Task inputKeys and outputKeys
This is to provide type safety for tasks and define a parameterized interface for task 
definitions such that tasks are completely re-usable within Conductor once 
registered. This provides a contract allowing the user to browse through available 
task definitions to use as part of their workflow where the tasks could have been 
implemented by another team/user. This feature would also involve enhancing the UI 
to display this contract.
Implementing MetadataDAO in Cassandra
As mentioned here, Cassandra module provides a partial implementation for 
persisting only the workflow executions. Metadata persistence implementation is not 
available yet and is something we are looking to add soon.
Pluggable Notifications on Task completion
Similar to the Workflow status listener, we would like to provide extensible 
interfaces for notifications on task execution.
Python client in Pypi
We have seen wide adoption of Python client within the community. However, 
there is no official Python client in Pypi, and lacks some of the newer additions 
to the Java client. We would like to achieve feature parity and publish a client 
from Conductor Github repository, and automate the client release to Pypi.
Removing Elasticsearch from critical path
While Elasticsearch is greatly useful in Conductor, we would like to make this 
optional for users who do not have Elasticsearch set-up. This means removing 
Elasticsearch from the critical execution path of a workflow and using it as an 
opt-in layer.
Pluggable authentication and authorization
Conductor doesn’t support authentication and authorization for API or UI, and 
is something that we feel would add great value and is a frequent request in 
the community.
Validations and Testing
Dry runs, i.e the ability to evaluate workflow definitions without actually running it 
through worker processes and all relevant set-up would make it much easier to test 
and debug execution paths.
If you would like to be a part of the Conductor community and contribute to one of the 
Wishlist items or something that you think would provide a great value add, please read 
through this guide for instructions or feel free to start a conversation on our Gitter 
channel, which is Conductor’s user forum.
We also highly encourage to polish, genericize and share any customizations that you may 
have built on top of Conductor with the community.
We really appreciate and are extremely proud of the community involvement, who have 
made several important contributions to Conductor. We would like to take this further and 
make Conductor widely adopted with a strong community backing.
Visit here for more details office setup today