Have a pipeline defined that utilizes a custom stage that had a specific purpose of parsing e-mails as the built-in one was insufficient. Also had a custom stage plugin that processed another kind of data. Customer decided to make one out of two; changed the bundle name and such to something new, so essentially a new plugin.
The next step was to modify the pipeline(s) that uses the old plugin to use the new one. An important desire was to maintain the metrics and the check-point of the task as the workflows has been running for some time. So didn't want to clear the task and start all over again.
The thought was to just change the pipeline and on the next run would start using the new plugin, but that didn't work too well. Customer reported that HCI got into a very unhappy state. The GUI was responding terribly slow and the tasks were not making much progress at all. Also getting a lot of failures reporting that the plugin was not found (referring to the old one no longer in the pipeline). Eventually, was able to navigate to the task and clear the task. After clearing the task things went back to operational.
Is there any best practices beyond "always clear the task if changing the pipeline", and is the hopeful behavior of being able to change pipeline and just pick it up the next run invalid?